---
source: raw/wiki-ingest-prompt.md
content_hash: 29f6a0914d428cf62eeb2a832d6b3048edf3e1d16aa7714c09f686377394d128
namespace: work
last_synced: 2026-05-29
confidence: 0.9
tags: [wiki, ingest, process, obsidian, automation]
---

# Wiki Ingest — Process

The wiki ingest job keeps `~/obsidian/wiki/` synchronised with source files referenced under `~/obsidian/raw/`. Each entry in `raw/` is a symlink to an external project file. The job runs as a launchd subprocess invoking `claude -p` (see [[personal-os-state-2026-04-27]] — needs local filesystem access, can't run in Hermes).

## Trigger

`personal.os.wiki-ingest` launchd agent fires daily at 22:00. Runs `~/scripts/run-wiki-ingest.sh`, which has a one-per-day guard.

## Algorithm

### Discover changes
For every file under `~/obsidian/raw/`:
1. Read file content (follows symlink).
2. Compute SHA256 via `shasum -a 256 <path>`.
3. Look in `~/obsidian/wiki/` for a `.md` file whose frontmatter has `source: raw/<filename>`.
4. If a wiki page exists AND its `content_hash` matches the freshly computed one — skip (up-to-date).
5. Otherwise — queue for synthesis.
6. Broken symlinks (target missing) — skipped entirely, no wiki page is created.

If queue empty → emit `Wiki is up to date. Nothing to ingest.` and stop.

### Synthesise each queued file
- Update existing wiki page if one references this source; otherwise create `~/obsidian/wiki/<topic-name>.md` (name by topic, not by source filename — e.g. `personal-os-schema.md`, not `schema.md`).
- Write a synthesis, **not** a verbatim copy. Extract facts, decisions, structure.
- Keep under 800 words per page; split large topics into linked pages using `[[double brackets]]`.

### Frontmatter contract
```yaml
---
source: raw/<filename>
content_hash: <sha256>
namespace: work
last_synced: <YYYY-MM-DD>
confidence: 0.8
tags: [tag1, tag2]
---
```

### Confidence levels
- `0.9` — highly structured, authoritative source (e.g. SQL schema)
- `0.8` — clear single source (default)
- `0.6` — inferred or partial content
- `1.0` — reserved for human-written notes; the ingester **must not** touch these

### Output report
```
Wiki Ingest — <date>

Processed: N files
Created: N new pages
Updated: N pages
Skipped: N (unchanged)

Pages updated:
- wiki/<filename> (source: raw/<source>)
```

## Rules (hard)

- Never copy source content verbatim — always synthesise.
- Never edit any `wiki/*.md` with `confidence: 1.0` — those are human-written.
- Never process files whose namespace is `family` unless explicitly instructed.
- Always update both `content_hash` and `last_synced` in frontmatter after writing.
- Always skip broken symlinks; never create a wiki page for a missing target.

## Why file-based, not DB-based

Earlier designs (see [[personal-os-state-2026-04-27]] Decision 8) had wiki-ingest read `file_references` + `wiki_pages` Postgres tables. That was dropped:
- Source of truth shifted to Obsidian vault filesystem.
- SHA256 in frontmatter gives incremental sync without DB dependency.
- Vault is self-contained — no DB schema migrations required.
- `obsidian-mcp` gives agents native vault access (Decision 2).

## Operational notes

- `personal.os.wiki-ingest` uses `claude -p` without `--allowedTools` — it may try unavailable tools; monitor logs (low-severity tech debt per state snapshot).
- Commit vault: `commit-vault.sh` runs nightly at 23:00, git-adds with LLM summary, pushes to NAS remote.
- NAS remote lives on ZFS data pool, not the boot pool, to survive TrueNAS updates: `ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git`.

## Key Points
- File-based incremental sync via SHA256 in frontmatter — no DB tables involved.
- One wiki page per source, named by topic.
- `confidence: 1.0` pages are off-limits to the ingester.
- Output report tells the user exactly what was processed/created/updated/skipped.
- Synthesis (~800 words), not transcription — split large sources.

## Related
[[personal-os-schema]] [[personal-os-state-2026-04-27]] [[personal-os-catchup-plan-2026-04-27]]