Wiki Ingest — Process

The wiki ingest job keeps ~/obsidian/wiki/ synchronised with source files referenced under ~/obsidian/raw/. Each entry in raw/ is a symlink to an external project file. The job runs as a launchd subprocess invoking claude -p (see personal-os-state-2026-04-27 — needs local filesystem access, can't run in Hermes).

Trigger

personal.os.wiki-ingest launchd agent fires daily at 22:00. Runs ~/scripts/run-wiki-ingest.sh, which has a one-per-day guard.

Algorithm

Discover changes

For every file under ~/obsidian/raw/:

Read file content (follows symlink).
Compute SHA256 via shasum -a 256 <path>.
Look in ~/obsidian/wiki/ for a .md file whose frontmatter has source: raw/<filename>.
If a wiki page exists AND its content_hash matches the freshly computed one — skip (up-to-date).
Otherwise — queue for synthesis.
Broken symlinks (target missing) — skipped entirely, no wiki page is created.

If queue empty → emit Wiki is up to date. Nothing to ingest. and stop.

Synthesise each queued file

Update existing wiki page if one references this source; otherwise create ~/obsidian/wiki/<topic-name>.md (name by topic, not by source filename — e.g. personal-os-schema.md, not schema.md).
Write a synthesis, not a verbatim copy. Extract facts, decisions, structure.
Keep under 800 words per page; split large topics into linked pages using [[double brackets]].

Frontmatter contract

---
source: raw/<filename>
content_hash: <sha256>
namespace: work
last_synced: <YYYY-MM-DD>
confidence: 0.8
tags: [tag1, tag2]
---

Confidence levels

0.9 — highly structured, authoritative source (e.g. SQL schema)
0.8 — clear single source (default)
0.6 — inferred or partial content
1.0 — reserved for human-written notes; the ingester must not touch these

Output report

Wiki Ingest — <date>

Processed: N files
Created: N new pages
Updated: N pages
Skipped: N (unchanged)

Pages updated:
- wiki/<filename> (source: raw/<source>)

Rules (hard)

Never copy source content verbatim — always synthesise.
Never edit any wiki/*.md with confidence: 1.0 — those are human-written.
Never process files whose namespace is family unless explicitly instructed.
Always update both content_hash and last_synced in frontmatter after writing.
Always skip broken symlinks; never create a wiki page for a missing target.

Why file-based, not DB-based

Earlier designs (see personal-os-state-2026-04-27 Decision 8) had wiki-ingest read file_references + wiki_pages Postgres tables. That was dropped:

Source of truth shifted to Obsidian vault filesystem.
SHA256 in frontmatter gives incremental sync without DB dependency.
Vault is self-contained — no DB schema migrations required.
obsidian-mcp gives agents native vault access (Decision 2).

Operational notes

personal.os.wiki-ingest uses claude -p without --allowedTools — it may try unavailable tools; monitor logs (low-severity tech debt per state snapshot).
Commit vault: commit-vault.sh runs nightly at 23:00, git-adds with LLM summary, pushes to NAS remote.
NAS remote lives on ZFS data pool, not the boot pool, to survive TrueNAS updates: ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git.

Key Points

File-based incremental sync via SHA256 in frontmatter — no DB tables involved.
One wiki page per source, named by topic.
confidence: 1.0 pages are off-limits to the ingester.
Output report tells the user exactly what was processed/created/updated/skipped.
Synthesis (~800 words), not transcription — split large sources.

personal-os-schema personal-os-state-2026-04-27 personal-os-catchup-plan-2026-04-27

4.0 KiB Raw Blame History