From 1c35fba9e29dbf98de4573640629eb3fcfa89022 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Mon, 27 Apr 2026 21:24:37 +0600 Subject: [PATCH] =?UTF-8?q?[2026-04-27]=20wiki/schema,=20wiki/ingest-proce?= =?UTF-8?q?ss=20=E2=80=94=20first=20ingest=20run?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- raw/schema.sql | 1 + raw/wiki-ingest-prompt.md | 1 + wiki/personal-os-schema.md | 92 +++++++++++++++++++++++++++++++++ wiki/wiki-ingest-process.md | 100 ++++++++++++++++++++++++++++++++++++ 4 files changed, 194 insertions(+) create mode 120000 raw/schema.sql create mode 120000 raw/wiki-ingest-prompt.md create mode 100644 wiki/personal-os-schema.md create mode 100644 wiki/wiki-ingest-process.md diff --git a/raw/schema.sql b/raw/schema.sql new file mode 120000 index 00000000..1f8889a0 --- /dev/null +++ b/raw/schema.sql @@ -0,0 +1 @@ +/Users/admin/Developer/personal-os/schema.sql \ No newline at end of file diff --git a/raw/wiki-ingest-prompt.md b/raw/wiki-ingest-prompt.md new file mode 120000 index 00000000..6c1be460 --- /dev/null +++ b/raw/wiki-ingest-prompt.md @@ -0,0 +1 @@ +/Users/admin/Developer/personal-os/agent/prompts/wiki-ingest.md \ No newline at end of file diff --git a/wiki/personal-os-schema.md b/wiki/personal-os-schema.md new file mode 100644 index 00000000..0ab9e5bb --- /dev/null +++ b/wiki/personal-os-schema.md @@ -0,0 +1,92 @@ +--- +source: raw/schema.sql +content_hash: 71b47e47b483834c887de14c76d5b16506d90ad64198d76a92a7b3731132dab8 +namespace: work +last_synced: 2026-04-27 +confidence: 0.9 +tags: [schema, postgres, database, asana, wiki] +--- + +# Personal OS Database Schema + +Postgres schema (`personal_os`) backing the Asana mirror, agent annotations, +file ingestion pipeline, semantic wiki, and memory store. Requires the +`vector` (pgvector) and `pg_trgm` extensions. Bootstrap with +`psql -U admin -d personal_os -f schema.sql`. + +## Asana mirror + +- **`tasks`** — one row per Asana task GID, upserted on each sync. + Stores due/start dates, completion, My Tasks section, assignee/creator, + primary project, timestamps, full `raw_json`, plus a `source` enum: + `my_tasks` (assigned to me) > `delegated` (I created, others assigned) + > `following` (CC'd) > `project` (project member, fallback). + Sync bookkeeping: `fetched_at`, `stories_fetched_before` cursor. +- **`stories`** — raw event log per task, keyed by Asana story GID. + `resource_subtype` covers comments, assignment changes, due-date edits, + section moves, dependency edits, attachments, completion, etc. +- **`task_edges`** — directed graph between tasks. `relation_type`: + `subtask`, `dependency` (blocked-by), `dependent` (blocking), + `project_sibling`, `mention` (referenced in a story). + `related_gid` may not yet exist in `tasks`. +- **`task_annotations`** — agent or user notes per (task, annotation_type). + Types: `irrelevant`, `watching`, `needs_action`, `snoozed` + (with `snoozed_until`). Never written by the fetcher. +- **`sync_state`** — one row per sync stream + (`my_tasks`, `following`, `delegated`, `project:{gid}`, + `workspace_events`). Tracks cursor, last sync, last full sync. +- **`task_embeddings`** — pgvector(1024) per task, HNSW index with + cosine ops; populated separately from sync. + +The **`active_tasks`** view filters out completed and irrelevant tasks +and tasks snoozed past today, ordered by source priority then due date +then modified-at. Starting point for daily review. + +## Phase 0 additions (2026-04-27) + +All core tables gain `namespace TEXT NOT NULL DEFAULT 'work'` so a single +DB can serve work/personal/family contexts. `tasks` also gets +`possibly_deleted` and `last_seen_in_full_sync` to track tasks that +disappear between full syncs without explicit deletion events. + +## File ingestion → wiki + +- **`file_references`** — files (via macOS security-scoped + `bookmark_data` + cached `last_known_path`) and web URLs + queued for wiki ingestion. Tracks `content_hash` (SHA256), + `mime_type`, `tags`, a `modification_log` JSONB, and a + `wiki_stale` flag the [[wiki-ingest-process]] consumes. + Namespace-checked (`work`/`personal`/`family`). +- **`wiki_pages`** — LLM-synthesised markdown, never a raw copy. + Has `sources` JSONB (file_ref/url/title), `confidence` float, + `superseded_by` self-FK for version chains, `stale` flag, + `last_synced_hash`, and a pgvector(1536) embedding for semantic + search. The optional ivfflat index is left commented; rebuild + once the table has 1000+ rows. + +## Memory store + +**`memory_store`** — semantic memory from Discord, Claude sessions, +and manual entries. `type` ∈ {fact, preference, decision, person}, +`source` ∈ {discord, claude, manual}, with entities JSONB, +confidence, optional `expires_at` (null = permanent), and a +pgvector(1536) embedding. Used by Hermes for cross-session context. + +## Indexing notes + +- Trigram GIN on `tasks.name` enables fuzzy task search. +- `tasks_due_on` is partial (only non-completed tasks). +- pgvector embedding indexes for `wiki_pages` and `memory_store` + are deferred until the tables have meaningful row counts. +- `task_embeddings` uses HNSW; the wiki/memory stores use ivfflat + (commented) — different recall/build trade-off per workload. + +## Key constraints + +- All `namespace` columns are CHECK-constrained to + `('work','personal','family')` on the new (Phase 0+) tables. +- Cascade deletes flow from `tasks` → `stories`, `task_edges`, + `task_annotations`, `task_embeddings`. + +## Related +[[personal-os-architecture]] [[wiki-ingest-process]] diff --git a/wiki/wiki-ingest-process.md b/wiki/wiki-ingest-process.md new file mode 100644 index 00000000..5e801b25 --- /dev/null +++ b/wiki/wiki-ingest-process.md @@ -0,0 +1,100 @@ +--- +source: raw/wiki-ingest-prompt.md +content_hash: 29f6a0914d428cf62eeb2a832d6b3048edf3e1d16aa7714c09f686377394d128 +namespace: work +last_synced: 2026-04-27 +confidence: 0.9 +tags: [wiki, ingest, agent, prompt] +--- + +# Wiki Ingest Process + +The agent prompt that maintains `~/obsidian/wiki/` as a synthesised +knowledge base over external project files exposed via symlinks in +`~/obsidian/raw/`. Run locally by `claude -p` (the launchd job +`run-wiki-ingest.sh` at 22:00) — not via API, because file-system +writes require a local Claude Code session. See +[[personal-os-architecture]] for where this fits in the broader system. + +## Discovery loop + +For every file in `~/obsidian/raw/`: + +1. Read the file (skip if the symlink target is missing). +2. Compute SHA256 with `shasum -a 256`. +3. Look for a wiki page whose frontmatter has `source: raw/`. +4. If a page exists and its `content_hash` already matches, skip. +5. Otherwise enqueue for synthesis. + +If nothing changed, the agent prints +"Wiki is up to date. Nothing to ingest." and exits. + +## Synthesis rules + +- Wiki pages are named by **topic**, not source filename + (e.g. `personal-os-schema.md`, not `schema.md`). +- Output is a **synthesis**, never a verbatim copy. Extract facts, + decisions, and structure. Hard cap of 800 words per page; split + into linked pages if the topic is larger. +- Use `[[double brackets]]` for cross-references between wiki pages. + +## Page format + +Frontmatter is the source of truth for incremental ingestion: + +```yaml +source: raw/ +content_hash: +namespace: work +last_synced: +confidence: 0.8 +tags: [tag1, tag2] +``` + +Body has a title, synthesis prose, a `## Key Points` list, and a +`## Related` line of `[[wiki-links]]`. + +## Confidence ladder + +- **1.0** — reserved for human-written notes in `work/`, `personal/`, + `family/`. The agent must never edit these files. +- **0.9** — highly structured / authoritative source (e.g. a schema). +- **0.8** — clear single source. Default. +- **0.6** — inferred or partial content. + +## Hard rules + +- Never copy source files verbatim — always synthesise. +- Never edit any file with `confidence: 1.0`. +- Never process files in `namespace: family` unless explicitly told. +- Always update `content_hash` and `last_synced` after writing a page. +- Skip broken symlinks; do not create a wiki page for them. + +## Reporting + +After the run, the agent emits a summary: + +``` +Wiki Ingest — + +Processed: N files +Created: N new pages +Updated: N pages +Skipped: N (unchanged) + +Pages updated: +- wiki/ (source: raw/) +``` + +## Key Points +- Hash-based incremental: re-runs are cheap when nothing changed. +- Topic-named pages decouple the wiki from source-file naming. +- Frontmatter `source` + `content_hash` is the ingestion contract; + the [[personal-os-schema]] mirrors the same idea in `wiki_pages`. +- Human-edited (confidence 1.0) pages are immutable to the agent — + the trust boundary between synthesis and curated knowledge. +- Local-only execution: writes to `~/obsidian/` need a real FS, + so the job runs under launchd via `claude -p`, not the API. + +## Related +[[personal-os-architecture]] [[personal-os-schema]]