98 lines
4.0 KiB
Markdown
98 lines
4.0 KiB
Markdown
---
|
|
source: raw/wiki-ingest-prompt.md
|
|
content_hash: 29f6a0914d428cf62eeb2a832d6b3048edf3e1d16aa7714c09f686377394d128
|
|
namespace: work
|
|
last_synced: 2026-05-29
|
|
confidence: 0.9
|
|
tags: [wiki, ingest, process, obsidian, automation]
|
|
---
|
|
|
|
# Wiki Ingest — Process
|
|
|
|
The wiki ingest job keeps `~/obsidian/wiki/` synchronised with source files referenced under `~/obsidian/raw/`. Each entry in `raw/` is a symlink to an external project file. The job runs as a launchd subprocess invoking `claude -p` (see [[personal-os-state-2026-04-27]] — needs local filesystem access, can't run in Hermes).
|
|
|
|
## Trigger
|
|
|
|
`personal.os.wiki-ingest` launchd agent fires daily at 22:00. Runs `~/scripts/run-wiki-ingest.sh`, which has a one-per-day guard.
|
|
|
|
## Algorithm
|
|
|
|
### Discover changes
|
|
For every file under `~/obsidian/raw/`:
|
|
1. Read file content (follows symlink).
|
|
2. Compute SHA256 via `shasum -a 256 <path>`.
|
|
3. Look in `~/obsidian/wiki/` for a `.md` file whose frontmatter has `source: raw/<filename>`.
|
|
4. If a wiki page exists AND its `content_hash` matches the freshly computed one — skip (up-to-date).
|
|
5. Otherwise — queue for synthesis.
|
|
6. Broken symlinks (target missing) — skipped entirely, no wiki page is created.
|
|
|
|
If queue empty → emit `Wiki is up to date. Nothing to ingest.` and stop.
|
|
|
|
### Synthesise each queued file
|
|
- Update existing wiki page if one references this source; otherwise create `~/obsidian/wiki/<topic-name>.md` (name by topic, not by source filename — e.g. `personal-os-schema.md`, not `schema.md`).
|
|
- Write a synthesis, **not** a verbatim copy. Extract facts, decisions, structure.
|
|
- Keep under 800 words per page; split large topics into linked pages using `[[double brackets]]`.
|
|
|
|
### Frontmatter contract
|
|
```yaml
|
|
---
|
|
source: raw/<filename>
|
|
content_hash: <sha256>
|
|
namespace: work
|
|
last_synced: <YYYY-MM-DD>
|
|
confidence: 0.8
|
|
tags: [tag1, tag2]
|
|
---
|
|
```
|
|
|
|
### Confidence levels
|
|
- `0.9` — highly structured, authoritative source (e.g. SQL schema)
|
|
- `0.8` — clear single source (default)
|
|
- `0.6` — inferred or partial content
|
|
- `1.0` — reserved for human-written notes; the ingester **must not** touch these
|
|
|
|
### Output report
|
|
```
|
|
Wiki Ingest — <date>
|
|
|
|
Processed: N files
|
|
Created: N new pages
|
|
Updated: N pages
|
|
Skipped: N (unchanged)
|
|
|
|
Pages updated:
|
|
- wiki/<filename> (source: raw/<source>)
|
|
```
|
|
|
|
## Rules (hard)
|
|
|
|
- Never copy source content verbatim — always synthesise.
|
|
- Never edit any `wiki/*.md` with `confidence: 1.0` — those are human-written.
|
|
- Never process files whose namespace is `family` unless explicitly instructed.
|
|
- Always update both `content_hash` and `last_synced` in frontmatter after writing.
|
|
- Always skip broken symlinks; never create a wiki page for a missing target.
|
|
|
|
## Why file-based, not DB-based
|
|
|
|
Earlier designs (see [[personal-os-state-2026-04-27]] Decision 8) had wiki-ingest read `file_references` + `wiki_pages` Postgres tables. That was dropped:
|
|
- Source of truth shifted to Obsidian vault filesystem.
|
|
- SHA256 in frontmatter gives incremental sync without DB dependency.
|
|
- Vault is self-contained — no DB schema migrations required.
|
|
- `obsidian-mcp` gives agents native vault access (Decision 2).
|
|
|
|
## Operational notes
|
|
|
|
- `personal.os.wiki-ingest` uses `claude -p` without `--allowedTools` — it may try unavailable tools; monitor logs (low-severity tech debt per state snapshot).
|
|
- Commit vault: `commit-vault.sh` runs nightly at 23:00, git-adds with LLM summary, pushes to NAS remote.
|
|
- NAS remote lives on ZFS data pool, not the boot pool, to survive TrueNAS updates: `ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git`.
|
|
|
|
## Key Points
|
|
- File-based incremental sync via SHA256 in frontmatter — no DB tables involved.
|
|
- One wiki page per source, named by topic.
|
|
- `confidence: 1.0` pages are off-limits to the ingester.
|
|
- Output report tells the user exactly what was processed/created/updated/skipped.
|
|
- Synthesis (~800 words), not transcription — split large sources.
|
|
|
|
## Related
|
|
[[personal-os-schema]] [[personal-os-state-2026-04-27]] [[personal-os-catchup-plan-2026-04-27]]
|