Writings ⇄ vault: the obsidian import/export subsystem

Updated · View the entry on sijie.xyz ↗

Today (landed): manual, owner-triggered, batch architecture. GET /api/admin/obsidian/export streams a standmeet-vault.zip (no temp file); POST /api/admin/obsidian/import accepts a whole-vault multipart upload (browser webkitdirectory, 200 MB cap — backend/internal/routes/admin/obsidian.go:149), returns a summary of {created, updated, skipped, errors}; GET /api/admin/obsidian/state reports when the last import ran (6cf4e2b08, 2026-08-21). This page covers the writings branch of that pipeline — writings are still a separate estate from the corpus tiers in addressing (flat, slug-keyed) and in linker (writing_refs) (confusables), even though since bfb8c7129 (2026-07-09) they are rows of the one corpus_notes table with genre='writing'. Wiki / subjectivity / raw / output go through SyncVault and export_corpus.go instead — see obsidian-sync-mechanism.

Zip layout and frontmatter contract

Export writes two directories:

  • writings/<slug>.md — each writing as frontmatter + body
  • attachments/<asset-id>.<ext> — deduped by storage key; extension guessed from content-type header

Frontmatter fields (the vault-to-writing schema):

  • title, slug, excerpt, tags, aliases — metadata
  • created, publish — timestamps and the import gate
  • cover_hue, cover_headline, cover_image — cover metadata
  • visibility, locked_body — permissions and locks

Import gate: publish: true is the ONLY path into the database. Anything else is silently counted as skipped. This is a deliberate boundary: the vault is the owner's draft or archived space; only explicitly published frontmatter matters.

Idempotency — and the web-edit guard that was removed

The idempotency key is obsidian_source_path — the vault-relative path to the .md file — with slug as the fallback (upsertFromVault, backend/internal/corpus/obsidian/import.go:252 / :279), so a moved file with a stable frontmatter slug matches its old row.

Re-import flow:

  • Path (or slug) exists in DB → UPDATE the row, overwrite from vault
  • Neither exists → CREATE a new row

The web-edit guard (updated_at > obsidian_imported_at + 1s → SKIP) was removed on 2026-07-15 (84080bac5, F-L-6): the vault is the single live source and sync makes destination equal source, no "who wins" (import.go:14). A web edit survives only if the owner exports it back before the next sync. SetObsidianMeta still stamps obsidian_imported_at = now after save, but as provenance, not as a guard.

flowchart TB
  F["incoming .md<br/>publish: true"] --> L{"row with same<br/>obsidian_source_path,<br/>else same slug?"}
  L -- no --> CR["CREATE"]
  L -- yes --> UP["UPDATE<br/>overwrite from vault<br/>(no web-wins since 84080bac5)"]
  CR --> ST["stamp imported_at"]
  UP --> ST

Attachment reference round-trip

Obsidian and StandMeet speak different reference languages. The system must bridge both directions losslessly (or at least predictably).

Export (writings → Obsidian):

  • Body contains standmeet-asset:uuid tokens
  • Rewrite to attachments/id.ext on export

Import (Obsidian → writings):

  • Incoming body has ![[img.png]] (image embeds) or ![alt](path) (markdown image links)
  • Basename resolution: match incoming reference against uploaded attachments by filename alone (e.g., img.png), independent of folder nesting. This follows Obsidian's own semantics for cross-vault file moves.
  • Pending UUID placeholder → standmeet-asset:uuid token (atomic with SaveWriting)
  • External http(s):// refs and unmatched refs are left as-is, not errored
flowchart LR
  subgraph EXPORT
    A1["body: standmeet-asset:uuid"] --> A2["rewrite → attachments/id.ext"]
  end
  subgraph IMPORT
    B1["body: ![[img.png]]<br/>![alt](path)"] --> B2["resolve by BASENAME<br/>against uploaded attachments"]
    B2 --> B3["pending-uuid → standmeet-asset:uuid<br/>atomic with SaveWriting"]
  end

Backlinks converge in two passes — fixpoint iteration

A write-once trap for forward references: the CrossRefs input on import is ignored entirely. Instead:

  1. SaveWriting is called with the body
  2. Inside SaveWriting, refreshCrossLinks re-parses literal [[X]] from the saved body
  3. It rebuilds writing_refs by resolving links only against writings that already exist in the database

Consequence: a forward link to a not-yet-imported writing stays unresolved on pass 1 and only connects on a second import (after both are created). The system is literally iterating refs = F(refs) to a fixed point — a deterministic, re-runnable operation that converges as more writings are imported.

sequenceDiagram
  participant V as vault upload
  participant S as SaveWriting
  participant R as writing_refs
  V->>S: import A (body has [[B]])
  S->>R: rebuild refs for A — B not found, link dangles
  V->>S: import B
  S->>R: rebuild refs for B
  Note over R: A→B STILL missing;<br/>fixpoint not reached
  V->>S: re-import A (pass 2)
  S->>R: rebuild refs for A — B now resolves
  Note over R: fixpoint reached

Be precise about what IS and ISN'T guaranteed: each file is idempotent (same file → same row, no duplicates), but the ref graph is not single-pass — importing the same ZIP a second time adds the forward links the first pass couldn't resolve. Deterministic in the limit (the fixpoint), not per run; the "import twice" wart is the user-visible face of exactly this.

Class view — the subsystem's moving parts

classDiagram
  class ExportDeps {
    Writings *corpus.WritingRepo
    Assets *corpus.AssetRepo
    Storage *storage.Client
    Corpus *corpus.VaultSyncRepo - corp notes, optional
  }
  class WriteZip {
    <<func>>
    takes ctx, ExportDeps, ownerID, io.Writer
    streams standmeet-vault.zip - no temp file
    rewrites standmeet-asset refs outbound
    then writeCorpusNotes when Corpus is set
  }
  class ImportVault {
    <<func>>
    takes ctx, WritingsTxDeps, MetaSetter, ownerID, []VaultFile
    returns ImportResult - created, updated, skipped, errors
    publish gate + source-path-or-slug upsert, no web-edit guard
  }
  class SaveWriting {
    <<func>>
    takes ctx, WritingsTxDeps, *SaveWritingInput
    returns (domain.Writing, error)
    atomic with asset token swap
  }
  class refreshCrossLinks {
    <<func>>
    takes ctx, deps, pgx.Tx, *domain.Writing
    re-parses literal wikilinks from the saved body
    resolves against EXISTING rows only
    rebuilds writing_refs - the fixpoint step
  }
  WriteZip ..> ExportDeps
  ImportVault --> SaveWriting : per accepted file
  SaveWriting --> refreshCrossLinks : inside the tx

Known gaps

Honest list of current limitations:

  1. Folder hierarchy lost — writings only now: export flattens nested writings/ folders into a single writings/ directory, and pickSlug derives a writing's slug from the bare basename. Since 9f1ffcf13 (2026-07-29) wiki/ · subjectivity/ · output/ are exported as genre folder + tree + folder-notes (export_corpus.go), so those round-trip; only the writings branch is still not folder-symmetric.

  2. Rename orphans rows — only when the slug changes too: identity is obsidian_source_path or slug (import.go:252 / :279; pinned by e2e/test/corpus-sync-rename.spec.ts, 82dd221aa, 2026-07-05). A moved or renamed file whose frontmatter slug is stable updates its old row. A rename that also changes the slug (no frontmatter slug — the filename is the slug) creates a new row, and the old one stays: the writings branch never deletes (import.go:65), unlike the corp branch's authoritative prune.

  3. Mid-stream errors not surfaced: export streams standmeet-vault.zip without buffering. If an error occurs mid-stream, the client receives a truncated zip and no error indication.

  4. Extension guessing is lossy: content-type headers are sniffed to guess file extensions. Some MIME types map to multiple extensions; the system picks one (e.g., .jpeg vs .jpg), and re-import may not match the original extension.

  5. cover_image doesn't render inline: the cover_image frontmatter field round-trips as plain YAML. Obsidian won't render it as an inline image preview, making the cover experience asymmetric between vault and web.

  6. Wiki and output tiers not wired — shipped: import via SyncVault (backend/internal/corpus/obsidian/sync.go:88), export via export_corpus.go (9f1ffcf13, 2026-07-29); see obsidian-sync-mechanism.

Both proposals landed — neither in the shape proposed here

The linker two-pass → whole-batch resolution

Proposed as two sequential passes (save bodies, then rebuild refs). What shipped is simpler: resolveLinks runs after all upserts, against the full owner-wide title index, so forward links resolve in one import. The "import twice" wart is gone, which was the point; the two-pass framing turned out to be an implementation detail of that goal rather than the goal.

Stable identity → keyed on title, and a rename deliberately orphans

Proposed as an exported id in frontmatter, resolved id > source_path > slug. Not what shipped, and the difference is a decision, not a shortfall. Identity is keyed on title (= the filename basename), owner-scoped and cross-genre — the vault's own check-links.sh already guarantees a basename is unique vault-wide, so the vault's constraint is the key.

Consequences, both deliberate: a move relocates the same row (including across genres — raw/x.md → wiki/x.md is the same note, which is exactly what made the semiotics promotion a move rather than a copy); a rename orphans, because the vault-side normalize-names already repoints wikilinks, so the vault is the place that owns renaming. obsidian_source_path / imported_at are kept as provenance (the web-wins guard they once served is gone, 84080bac5); source_path becomes the identity only when a basename is duplicated across the corpus — then the same-name files keep their own rows (F-L-2 / F-L-61, sync.go:4-7, sync_ambiguity.go).

Writing an id into the owner's own markdown was rejected for the reason it usually is: it puts a database's bookkeeping into a file a human edits by hand.

Cross-references

Related notes