[2026-04-27] wiki/schema, wiki/ingest-process — first ingest run

wiki/personal-os-schema.md

@@ -0,0 +1,92 @@
+---
+source: raw/schema.sql
+content_hash: 71b47e47b483834c887de14c76d5b16506d90ad64198d76a92a7b3731132dab8
+namespace: work
+last_synced: 2026-04-27
+confidence: 0.9
+tags: [schema, postgres, database, asana, wiki]
+---
+
+# Personal OS Database Schema
+
+Postgres schema (`personal_os`) backing the Asana mirror, agent annotations,
+file ingestion pipeline, semantic wiki, and memory store. Requires the
+`vector` (pgvector) and `pg_trgm` extensions. Bootstrap with
+`psql -U admin -d personal_os -f schema.sql`.
+
+## Asana mirror
+
+- **`tasks`** — one row per Asana task GID, upserted on each sync.
+  Stores due/start dates, completion, My Tasks section, assignee/creator,
+  primary project, timestamps, full `raw_json`, plus a `source` enum:
+  `my_tasks` (assigned to me) > `delegated` (I created, others assigned)
+  > `following` (CC'd) > `project` (project member, fallback).
+  Sync bookkeeping: `fetched_at`, `stories_fetched_before` cursor.
+- **`stories`** — raw event log per task, keyed by Asana story GID.
+  `resource_subtype` covers comments, assignment changes, due-date edits,
+  section moves, dependency edits, attachments, completion, etc.
+- **`task_edges`** — directed graph between tasks. `relation_type`:
+  `subtask`, `dependency` (blocked-by), `dependent` (blocking),
+  `project_sibling`, `mention` (referenced in a story).
+  `related_gid` may not yet exist in `tasks`.
+- **`task_annotations`** — agent or user notes per (task, annotation_type).
+  Types: `irrelevant`, `watching`, `needs_action`, `snoozed`
+  (with `snoozed_until`). Never written by the fetcher.
+- **`sync_state`** — one row per sync stream
+  (`my_tasks`, `following`, `delegated`, `project:{gid}`,
+  `workspace_events`). Tracks cursor, last sync, last full sync.
+- **`task_embeddings`** — pgvector(1024) per task, HNSW index with
+  cosine ops; populated separately from sync.
+
+The **`active_tasks`** view filters out completed and irrelevant tasks
+and tasks snoozed past today, ordered by source priority then due date
+then modified-at. Starting point for daily review.
+
+## Phase 0 additions (2026-04-27)
+
+All core tables gain `namespace TEXT NOT NULL DEFAULT 'work'` so a single
+DB can serve work/personal/family contexts. `tasks` also gets
+`possibly_deleted` and `last_seen_in_full_sync` to track tasks that
+disappear between full syncs without explicit deletion events.
+
+## File ingestion → wiki
+
+- **`file_references`** — files (via macOS security-scoped
+  `bookmark_data` + cached `last_known_path`) and web URLs
+  queued for wiki ingestion. Tracks `content_hash` (SHA256),
+  `mime_type`, `tags`, a `modification_log` JSONB, and a
+  `wiki_stale` flag the [[wiki-ingest-process]] consumes.
+  Namespace-checked (`work`/`personal`/`family`).
+- **`wiki_pages`** — LLM-synthesised markdown, never a raw copy.
+  Has `sources` JSONB (file_ref/url/title), `confidence` float,
+  `superseded_by` self-FK for version chains, `stale` flag,
+  `last_synced_hash`, and a pgvector(1536) embedding for semantic
+  search. The optional ivfflat index is left commented; rebuild
+  once the table has 1000+ rows.
+
+## Memory store
+
+**`memory_store`** — semantic memory from Discord, Claude sessions,
+and manual entries. `type` ∈ {fact, preference, decision, person},
+`source` ∈ {discord, claude, manual}, with entities JSONB,
+confidence, optional `expires_at` (null = permanent), and a
+pgvector(1536) embedding. Used by Hermes for cross-session context.
+
+## Indexing notes
+
+- Trigram GIN on `tasks.name` enables fuzzy task search.
+- `tasks_due_on` is partial (only non-completed tasks).
+- pgvector embedding indexes for `wiki_pages` and `memory_store`
+  are deferred until the tables have meaningful row counts.
+- `task_embeddings` uses HNSW; the wiki/memory stores use ivfflat
+  (commented) — different recall/build trade-off per workload.
+
+## Key constraints
+
+- All `namespace` columns are CHECK-constrained to
+  `('work','personal','family')` on the new (Phase 0+) tables.
+- Cascade deletes flow from `tasks` → `stories`, `task_edges`,
+  `task_annotations`, `task_embeddings`.
+
+## Related
+[[personal-os-architecture]] [[wiki-ingest-process]]

wiki/wiki-ingest-process.md

@@ -0,0 +1,100 @@
+---
+source: raw/wiki-ingest-prompt.md
+content_hash: 29f6a0914d428cf62eeb2a832d6b3048edf3e1d16aa7714c09f686377394d128
+namespace: work
+last_synced: 2026-04-27
+confidence: 0.9
+tags: [wiki, ingest, agent, prompt]
+---
+
+# Wiki Ingest Process
+
+The agent prompt that maintains `~/obsidian/wiki/` as a synthesised
+knowledge base over external project files exposed via symlinks in
+`~/obsidian/raw/`. Run locally by `claude -p` (the launchd job
+`run-wiki-ingest.sh` at 22:00) — not via API, because file-system
+writes require a local Claude Code session. See
+[[personal-os-architecture]] for where this fits in the broader system.
+
+## Discovery loop
+
+For every file in `~/obsidian/raw/`:
+
+1. Read the file (skip if the symlink target is missing).
+2. Compute SHA256 with `shasum -a 256`.
+3. Look for a wiki page whose frontmatter has `source: raw/<filename>`.
+4. If a page exists and its `content_hash` already matches, skip.
+5. Otherwise enqueue for synthesis.
+
+If nothing changed, the agent prints
+"Wiki is up to date. Nothing to ingest." and exits.
+
+## Synthesis rules
+
+- Wiki pages are named by **topic**, not source filename
+  (e.g. `personal-os-schema.md`, not `schema.md`).
+- Output is a **synthesis**, never a verbatim copy. Extract facts,
+  decisions, and structure. Hard cap of 800 words per page; split
+  into linked pages if the topic is larger.
+- Use `[[double brackets]]` for cross-references between wiki pages.
+
+## Page format
+
+Frontmatter is the source of truth for incremental ingestion:
+
+```yaml
+source: raw/<filename>
+content_hash: <sha256>
+namespace: work
+last_synced: <YYYY-MM-DD>
+confidence: 0.8
+tags: [tag1, tag2]
+```
+
+Body has a title, synthesis prose, a `## Key Points` list, and a
+`## Related` line of `[[wiki-links]]`.
+
+## Confidence ladder
+
+- **1.0** — reserved for human-written notes in `work/`, `personal/`,
+  `family/`. The agent must never edit these files.
+- **0.9** — highly structured / authoritative source (e.g. a schema).
+- **0.8** — clear single source. Default.
+- **0.6** — inferred or partial content.
+
+## Hard rules
+
+- Never copy source files verbatim — always synthesise.
+- Never edit any file with `confidence: 1.0`.
+- Never process files in `namespace: family` unless explicitly told.
+- Always update `content_hash` and `last_synced` after writing a page.
+- Skip broken symlinks; do not create a wiki page for them.
+
+## Reporting
+
+After the run, the agent emits a summary:
+
+```
+Wiki Ingest — <date>
+
+Processed: N files
+Created: N new pages
+Updated: N pages
+Skipped: N (unchanged)
+
+Pages updated:
+- wiki/<filename> (source: raw/<source>)
+```
+
+## Key Points
+- Hash-based incremental: re-runs are cheap when nothing changed.
+- Topic-named pages decouple the wiki from source-file naming.
+- Frontmatter `source` + `content_hash` is the ingestion contract;
+  the [[personal-os-schema]] mirrors the same idea in `wiki_pages`.
+- Human-edited (confidence 1.0) pages are immutable to the agent —
+  the trust boundary between synthesis and curated knowledge.
+- Local-only execution: writes to `~/obsidian/` need a real FS,
+  so the job runs under launchd via `claude -p`, not the API.
+
+## Related
+[[personal-os-architecture]] [[personal-os-schema]]