Wiki Ingest — Process

Daily process that keeps ~/obsidian/wiki/ synchronised with external source files surfaced via symlinks in ~/obsidian/raw/. Runs as a launchd subprocess (personal.os.wiki-ingest) invoking claude -p — not a Hermes cron job, because it needs local filesystem access to read sources and write wiki pages.

Flow

STEP 0 — Discover changed files

For each entry in ~/obsidian/raw/:

Read the symlink target.
Compute shasum -a 256.
Scan ~/obsidian/wiki/ for any .md whose frontmatter has source: raw/<filename>.
If page exists AND content_hash matches → skip (unchanged).
Otherwise → queue for processing.
If a symlink is broken, skip silently — never create a page for missing content.

Empty queue → print "Wiki is up to date. Nothing to ingest." and exit.

STEP 1 — Synthesise (per file)

Update an existing wiki page if one references the source, otherwise create ~/obsidian/wiki/<topic-name>.md named by topic, not source filename (e.g. personal-os-schema.md, not schema.md).
Write a synthesis: extract key facts, decisions, structure — never copy verbatim.
Cap at ~800 words; split big topics into linked pages.
Use [[wiki-links]] for cross-references.

STEP 2 — Report

Print processed / created / updated / skipped counts and a list of touched pages.

Frontmatter Schema

source: raw/<filename>
content_hash: <sha256>
namespace: work
last_synced: <YYYY-MM-DD>
confidence: 0.8
tags: [tag1, tag2]

content_hash and last_synced are always updated after a write — they are how the next ingest decides what changed.

Confidence Scale

Value	Meaning
0.6	inferred or partial content
0.8	clear single source (default)
0.9	highly structured, authoritative source (e.g. SQL schema)
1.0	reserved for human-written notes in `work/`, `personal/`, `family/`

Hard Rules

Never verbatim-copy source files; always synthesise.
Never edit files marked confidence: 1.0 — those are human-authored.
Never process namespace: family files unless explicitly told.
Always update content_hash and last_synced after writing.
Broken symlink → skip, do not create.

Why File-Based, Not DB-Backed

Earlier design (v3) had wiki_pages / file_references tables drive ingest via SQL. That was replaced by file-based ingest because obsidian-mcp now exposes the vault directly — the filesystem is the source of truth and the DB tables would have been a duplicate index of the same content. The Postgres tables for personal-os-schema still exist but the agent doesn't read or write them anymore (see personal-os-state-2026-04-27).

personal-os-schema personal-os-state-2026-04-27 personal-os-catchup-plan-2026-04-27

3.0 KiB Raw Permalink Blame History