Wiki Ingest Process

The wiki-ingest agent maintains ~/obsidian/wiki/ by synthesising markdown pages from source files exposed as symlinks under ~/obsidian/raw/. It is a local claude -p subprocess invoked nightly by the personal.os.wiki-ingest launchd agent (22:00 daily) — see personal-os-state-2026-04-27 for the runtime layout.

Inputs

Each entry under ~/obsidian/raw/ is a symlink to an external project file. The wiki keeps a synthesised page per source, identified by a frontmatter source: raw/<filename> line and tracked for staleness via a content_hash (SHA256) in the same frontmatter.

Workflow

Step 0 — Discover changes

For every file in ~/obsidian/raw/:

Read the file content.
Compute SHA256 (shasum -a 256 <path> | awk '{print $1}').
Search ~/obsidian/wiki/ for any .md whose frontmatter has source: raw/<filename>.
If a page exists and content_hash matches → skip.
Otherwise add to the work queue.

If the queue is empty, output Wiki is up to date. Nothing to ingest. and stop.

Step 1 — Process each changed file

Find or create: update the existing page if found, otherwise create ~/obsidian/wiki/<topic-name>.md. Filenames follow the topic, not the source filename — e.g. personal-os-schema.md, not schema.md.
Synthesise: extract key facts, decisions, and structure. Never copy the source verbatim. Keep each page under 800 words; split into linked pages if the topic is too large.
Cross-reference: use [[double brackets]] to link related wiki pages.

Step 2 — Report

After processing, print a summary with counts of created, updated, and skipped pages, followed by the list of pages touched and their sources.

Frontmatter contract

source: raw/<filename>
content_hash: <sha256>
namespace: work
last_synced: <YYYY-MM-DD>
confidence: 0.8
tags: [tag1, tag2]

confidence is a coarse trust signal:

Value	Meaning
1.0	Reserved for human-written notes (`work/`, `personal/`, `family/`).
0.9	Highly structured, authoritative source.
0.8	Default for clear single-source synthesis.
0.6	Inferred or partial content.

Hard rules

Never copy source files verbatim — always synthesise.
Never edit a wiki page with confidence: 1.0 — those are human-authored.
Never process files belonging to namespace: family unless explicitly instructed.
Always update content_hash and last_synced after rewriting a page.
If a symlink in raw/ is broken (target missing), skip it — do not create a stub wiki page.

Why this lives in the prompt, not a DB

Per Decision 8 in personal-os-state-2026-04-27, the original v3 design used file_references + wiki_pages Postgres tables. That was replaced with file-only state: SHA256 in frontmatter is enough for incremental sync, the vault is self-contained and portable, and Obsidian (via the obsidian-mcp package) becomes the single source of truth. The DB tables still exist in personal-os-schema but are unused in the live flow.

Operational notes

Runs once a day under launchd, so the prompt is allowed to be slow.
Invoked without --allowedTools in v1 (tracked as low-severity tech debt in personal-os-state-2026-04-27 §6). The agent may try to call tools it doesn't have — monitor stderr.
The commit-vault.sh script (23:00 daily) picks up any new wiki pages and pushes them to the NAS git remote.

Key Points

Source-of-truth is the filesystem; SHA256 in YAML frontmatter is the staleness signal.
Filenames follow topic, not source filename.
Synthesis (≤ 800 words), never verbatim.
Confidence 1.0 pages are human-owned and untouchable.
Family namespace requires explicit user opt-in per run.

personal-os-state-2026-04-27 personal-os-schema personal-os-catchup-plan-2026-04-27 ddg-asana-workflow

4.2 KiB Raw Blame History