obsidian sync mechanism

Updated · View the entry on sijie.xyz ↗

Bidirectional sync between StandMeet and an Obsidian vault. Shaped like Quartz / obsidian-importer: two buttons, each a batch job, owner-triggered manually — no file watcher, no live sync.

Endpoints (backend/internal/routes/admin/obsidian.go:152)

  • GET /api/admin/obsidian/export → a zip (writings/<slug>.md + attachments/<id>.<ext>, plus wiki/… · subjectivity/… · output/… as genre folder + tree); each note is rendered as frontmatter + body
  • POST /api/admin/obsidian/import → multipart upload of the whole vault (browser webkitdirectory, 200 MB cap); each file carries its vault-relative path
  • GET /api/admin/obsidian/state → when the last import ran (obsidian_state.go, 6cf4e2b08, 2026-08-21) — before this the screen had no past tense at all

Sync pipeline (backend/internal/corpus/obsidian/sync.go · SyncVault)

Import is one batch pass over the uploaded vault. classifyVault (sync_classify.go:65) splits files by top-level folder into three buckets — corp, writing, css (hidden paths and _templates filtered, except .obsidian CSS which is harvested). raw/ is no longer its own bucket: since d925d9081 (2026-07-09) it folds into corp as genre='raw', frontmatter-exempt (whole file = body), and goes through the same node-tree materializer. The corp bucket additionally runs the tree → ambiguity check → reconcile → link-resolution → prune stages. This flowchart is the authoritative overview; the prose below details each stage.

flowchart TD
  U["POST /obsidian/import<br/>whole vault (webkitdirectory);<br/>each file carries its vault-relative path"] --> C["classifyVault(files)<br/>hidden + _templates filtered (except .obsidian CSS)<br/>top folder → bucket"]
  C -->|"writing/ (or writings/)"| W["syncWritings → Writings.ImportWritings<br/>.md + attachments · publish-gated · writing_refs"]
  C -->|".obsidian snippets + appearance.json"| S["syncCSS<br/>harvest owner CSS config"]
  C -->|"genre = wiki / subjectivity / raw"| T["corp bucket · vaultNote list<br/>parseCorpNote: tolerant frontmatter<br/>(raw: fm-exempt, whole file = body)"]
  T --> BT["buildDesiredTree(corp)<br/>folder nesting → node tree (parent_id)<br/>folder-note collapse: foo/foo.md ⇒ the foo node"]
  BT --> AM["ambiguousTitles (sync_ambiguity.go)<br/>ask the CORPUS which basenames are duplicated;<br/>unanswerable → abort the whole batch"]
  AM --> RN["reconcileNode per node (parent-first)<br/>claim by title (cross-genre, supports move);<br/>duplicated title → claim by source_path<br/>unchanged → skip · else create / update<br/>record id → child parenting + link index"]
  RN --> RL["resolveLinks(tree)<br/>whole-batch: owner title + aliases index → note_refs edges<br/>(forward links after all nodes exist;<br/>backlinks = reverse query over note_refs)"]
  RL --> PR["pruneAbsent (sync_prune.go)<br/>authoritative mode: delete vault-imported notes<br/>absent from this batch · partial mode: never delete"]
  PR --> RES["ImportResult { created, updated, skipped, deleted, errors }"]

Reverse direction (export) is a separate batch — a node with children flattens back to foo/foo.md + children under foo/ (export_corpus.go); writings go out flat as writings/<slug>.md; see writings-import-export.

Writing-branch import detail (backend/internal/corpus/obsidian/import.go)

  1. Partition .md vs non-.md; attachments are indexed by basename (matches how Obsidian resolves [[image.png]] by basename regardless of subfolder)
  2. For each .md:
    • Parse frontmatter; publish != true → skip outright (import.go:146 — the writings branch is still ingest-gated; the corp branch is not, see the invariants below)
    • Image refs in the body → look up bytes in the attachment index → mint pending-<uuid>, rewrite the body ref to standmeet-asset:pending-<uuid>
    • Find an existing writing by obsidian_source_path, else by slug (upsertFromVault, import.go:252 / :279) → update; none → create. There is no web-wins guard any more (84080bac5, 2026-07-15, F-L-6): the vault is the single live source; a web edit is kept only by exporting it back before the next sync
    • SetObsidianMeta stamps imported_at
  3. Return { created, updated, skipped, errors } (writings never delete)

Folder notes (convention A) — must be specially recognized

The vault represents a node that has children as a folder foo/ whose own content lives in foo/foo.md (the folder note; enforced by a pre-commit hook). The sync must special-case this so the folder note maps to the node itself, not to a duplicate child:

  • Rule: if basename(file, ".md") == basename(dirname(file)), the file is a folder note → its node path = the directory path (a/b/foo), not the naive filesystem derivation a/b/foo/foo.
  • So market/market.md ⇒ the market node (path …/market); a sibling market/two-scarcities-arbitrage.md ⇒ a child of market (parent_id → market). One node per folder, no duplicate foo/foo slug.
  • Inverse on export: a node with children is written as foo/foo.md + its children under foo/.

Implemented for the corp path

buildDesiredTree (sync_tree.go) collapses folder notes for the wiki/subjectivity sync — market/market.md becomes the market node, siblings become its children (parent_id), no duplicate foo/foo. Still open: the writings branch derives slug as plain basename(path) (pickSlug, see protocol) and does not collapse folder notes yet.

Key invariants

  • publish is a visibility gate, not an ingest gate (F-L-8, 84080bac5, 2026-07-15; sync.go:13): every routed corp .md persists regardless; publish only feeds the DB's published = visible to anonymous visitors / in the sitemap; published=false means "needs a code", not "doesn't exist". An absent publish key keeps the current value — silence is not a no (F-L-22, 8bb2e1c00, 2026-08-08; keepPublish, sync.go:229). The writings branch alone still skips publish != true outright
  • The vault is the single live source (F-L-6, replaces "web edits win"): sync makes destination equal source, no timestamp comparison; an authoritative (whole-vault) sync prunes vault-imported notes the vault no longer has (sync_prune.go), a partial upload never deletes
  • Body [[slug]]/[[Title]] → edge tables: link relationships come from the body, not frontmatter — corp notes write note_refs (resolved by title and frontmatter aliases), writings write writing_refs
  • Which fields come from the template vs. from elsewhere → see protocol

Related notes