Connector plugins — installable, consumer-agnostic

Updated · View the entry on sijie.xyz ↗

A connector is NOT MCP — it speaks category contracts, not the MCP wire protocol; the two plugin axes touch only at connector-deps (confusables, incl. action vs sync mode).

Notation:

  • Consumer = agent, platform features, future IM-gateway / job-loop
  • Category Contract = platform-owned interface (e.g. CalendarContract, MailContract)
  • Connector Binding = declarative mapping (YAML) of provider operations → category contract
  • Protocol kind = built-in Go implementation (SMTP/CalDAV, plus the contract-less Telegram token vault; no IMAP or LDAP impl exists in backend/internal/connector/)
  • OpenAPI kind = author-supplied OpenAPI spec + binding (per-SaaS HTTP)

Definition: credentialed external integration

A connector is a credentialed external integration, consumer-agnostic, structured exactly like a WordPress plugin. Anyone can author one; an owner installs it into their instance. The Hub is the neutral base; we are not bundling a Nango catalog.

Key property: the consumer (agent, platform feature, future gateway) never holds the credential. Credentials stay sealed in the connector layer.

Two kinds (transparently interchangeable)

A category (e.g. calendar, mail) can be satisfied by either kind:

  • openapi — HTTP-based SaaS. Author supplies OpenAPI 3.0 spec + binding YAML. Credential form is derived from securitySchemes (oauth2, apiKey, http basic, bearer); owner picks in UI when multiple present.
  • protocol — Standard protocol (SMTP, CalDAV; Telegram since 2026-09-04, 8c819f26d — protocol_telegram.go, category im, token vault only, consumed by im-bridge not by a category contract). Built-in Go implementation; fixed credential descriptor. One impl covers the long tail.

Both have the same plugin shape and deployment path.

Three-layer architecture

① Category Contract (platform-owned, fixed)
Defines what operations a category must support. E.g.:

  • CalendarContract: list_busy(range) → bool grid, create_event(title, when, attendees) → event, cancel_event(id)
  • MailContract: send(to, subject, body, attachments) → message ID

② Connector Binding (declarative YAML)
Maps the contract onto the provider's operations. For OpenAPI: contract method → operationId, with JSONata transforms on request + response. For protocol: direct impl.

③ Generic runtime (contract call → result)
OpenAPI: call → token injection → HTTP call → normalize response. Protocol: direct protocol client call.

All three layers are independent of any particular provider. The kernel holds zero provider-specific code (google-calendar / smtp never appear in host).

Fixed decisions (design lock)

  • Mapping language: JSONata only (AWS Step Functions precedent).
  • OpenAPI version: 3.0 only.
  • Multi-auth: When a provider defines multiple securityScheme, the owner picks one in the UI at connect time.
  • Built-ins shape: Built-in connectors (shipped in-repo) have the identical shape as uploaded connectors.

Code reality

File layout:

  • backend/internal/connector/ — Hub, category slots, OpenAPI spec/binding/runtime/ingest, protocol_* implementations, builtins/data
  • connector.Service (service.go + svc_*.go; the connectorsvc package was merged in on 2026-07-26, 1bc9ba8b0) — credentials (AES-GCM via cryptobox), connect, oauth, activate, disconnect
  • capreg/depresolver.go — manifest Requires:["calendar"] → named dependency providers; unmet → capability hidden, fail-closed
  • Retry policy per-call-class
  • Plugins receive a call HANDLE, never raw credentials

Credential handling:
Encrypted at rest, decrypted only inside connector package. Injected per request via AuthInjector closure into *http.Request. Outside the package, callers see only Connected boolean + result data.

Class view — contracts on top, kinds below

classDiagram
  class CalendarProxy {
    <<interface - connector/contract, consumer-facing>>
    +Connected(ctx, ownerID) (bool, error)
    +FreeBusy(ctx, ownerID, FreeBusyReq) ([]BusyInterval, error)
    +InsertEvent(ctx, ownerID, *InsertEventReq) (InsertedEvent, error)
    +DeleteEvent(ctx, ownerID, eventID, attendeeEmail) error
  }
  class MailProxy {
    <<interface - connector/contract, consumer-facing>>
    +Connected(ctx, ownerID) (bool, error)
    +Send(ctx, ownerID, MailMessage) (MailReceipt, error)
  }
  class MailMessage {
    To, Subject string
    Body, HTML string
  }
  class Connector {
    <<interface - hub-facing>>
    +Name() string
    +Kind() string
    +Connected(ctx, ownerID) (bool, error)
  }
  class Hub {
    -conns map[string]Connector
    +Register(c) / Upsert(c)
    +Resolve(name) (Connector, bool)
  }
  class openapiRuntime {
    -spec *Spec
    -binding *Binding - JSONata
    -doer Doer
    -baseURL string
    +Call(ctx, op, input, dst, AuthInjector) error
  }
  class AuthInjector {
    <<func type>>
    func(req *http.Request) error
    creds sealed - built inside connector pkg
  }
  class protocolImpl {
    built-in Go clients
    SMTP, CalDAV
  }
  CalendarProxy <|.. openapiRuntime : adapter
  CalendarProxy <|.. protocolImpl : adapter
  MailProxy <|.. openapiRuntime : adapter
  MailProxy <|.. protocolImpl : adapter
  MailProxy ..> MailMessage
  Hub o-- Connector : by name
  openapiRuntime ..> AuthInjector : per call

Two facings, deliberately split: consumers hold the contract proxies (CalendarProxy/MailProxy in backend/internal/connector/contract/ — category verbs, no provider anywhere; Send returns a MailReceipt carrying the provider's message id); the Hub tracks Connectors (name/kind/connected — lifecycle only). Which kind satisfies a proxy is invisible to consumers. This is already the Bridge/Strategy shape; per the judgment-audit rule, no further pattern needed.

Red-contract test suite (BUILT: 66 connector-*.spec.ts files at 36789537d, 2026-09-07)

Each region pinned by executable red tests:

  • Ingest — spec parsing, binding validation
  • Cred-form derivation — securitySchemes → UI form schema
  • JSONata binding — transforms request + response
  • Connect flow — OAuth refresh, credential storage
  • Protocol SMTP — built-in SMTP client (mailer_smtp.go; there is no IMAP client)
  • Consumption loop — consumer calling contract methods
  • Upload mgmt — owner upload, lifecycle, deletion
  • Security — SSRF reject intranet servers, cred non-leak, per-owner isolation (overlaps connector-egress-guard)

Design decisions pinned by red tests:

  • One active connector per category slot
  • Agent-tool exposure opt-in per operation (op_<operationId>, per-op ACL)
  • Disconnect retains credentials
  • External $ref rejected (no cross-file schema composition)

Two modes: action (proxy) and sync (ingest)

Action (proxy) — synchronous consumption. Consumers call the connector; it proxies to the provider and returns result. This is the main path (agent tools, platform features).

Sync (ingest) — asynchronous ingestion. The connector fetches data on a schedule (or event-triggered). Obsidian vault sync is a sync-mode connector now (NewSyncConnector + SyncIngester); the seam where the corpus pillar meets this one is closed.

Status (July 2026, re-checked 2026-09-07 at 36789537d): landed

The proxy layer landed earlier; the installable, consumer-agnostic design has now landed too — the owner-upload flow is real code (Repo.SaveUploaded/UpdateUploaded in connection_repo_uploaded.go, Service.UpdateUploaded in svc_manage.go), the red-contract suite grew to 66 e2e spec files (deps/retry/security/upload/ingest/binding/matrix), the earlier 13-test.fixme tail has since been converted to live tests, and the TODO(impl) mock-infrastructure gaps were closed on 2026-07-03 (059dc5c13) — 0 remain; post-landing refinements continue (the assemble redesign removed the provider dropdown; the generic-connector-shape refactor; credform derives from authform; PKCE on the OAuth dance, e23c0c9f4, 2026-08-20). Sync-mode (the Obsidian seam) shipped on 2026-07-08 (d51805372, backend/internal/connector/sync.go).

Related notes