[2026-06-01] personal-os schema, state, catchup plan, wiki ingest process

wiki/personal-os-catchup-plan-2026-04-27.md

@@ -0,0 +1,125 @@
+---
+source: raw/personal-os-catchup-plan-2026-04-27.md
+content_hash: 5081ceb719104e1b7c6b5edd642c1daf38b3fd7cf1a96442a00516e7cdb15379
+namespace: work
+last_synced: 2026-06-01
+confidence: 0.8
+tags: [personal-os, planning, executor, catchup]
+---
+
+# Personal OS — Catch-Up Plan (2026-04-27)
+
+Handoff doc for the executor agent: what to build next and why. Companion
+to [[personal-os-state-2026-04-27]] (state snapshot). All open questions
+resolved at write time; postponed: NAS migration, personal, Phase 2-family.
+
+## Architectural Principle
+
+**No new YAML files, no new shell scripts, no new launchd plists.**
+Everything extends existing components: postgres tables,
+`generate-status.js` queries, prompt files in `agent/prompts/`, Eagle's
+SOUL.md rules. The system has enough moving parts.
+
+## Nine Deliverables (Dependency Order)
+
+### 1. Schema migration
+Three additions to [[personal-os-schema]]:
+- `corrections_log` — Tactic records pushback by `reason_tag`,
+  Retrospector aggregates patterns. Table beats markdown because
+  Retrospector needs `GROUP BY`.
+- `executor_runs` — state machine for autonomous executor work
+  (`analyzing → awaiting_analysis_approval → fixing → pr_open →
+  ci_running → ci_failed → awaiting_user → complete`). Survives
+  worker crashes mid-run.
+- `activity_daily.quadrant` column — `project | aor | career |
+  strategy | other` for Retrospector drift detection.
+
+### 2. `generate-status.js` — three new query blocks
+- **Advisor pipeline**: last 14d advisor-tagged tasks → new `## ⚡ Career`
+  section. Empty result → omit (no noise).
+- **Visibility gaps**: open Assessment / O-N / O-L tasks where Alex
+  hasn't commented in 7+ days → `## 📊 Visibility` section.
+- **Quadrant classification at AW write time**: map project → quadrant
+  using a static map plus a *dynamic* "current top priority" slot read
+  from config so it shifts as focus shifts.
+
+### 3. `inbox-triage.md` STEP 0
+Gating + scope filter at prompt level (not wrapper) so the agent can
+override on edge cases:
+- Active calendar meeting → silent exit.
+- `focus_score > 60` + Xcode/Cursor + 20+ min uninterrupted → silent
+  unless 🔴 section is 3h+ stale.
+- Scope: only **new** signal (new human comments, new assignments,
+  Ship/TD status changes, new mentions). **Never** overdue tasks
+  (those belong in `daily-brief`, not `#inbox`).
+
+### 4. `retrospector.md` — SQL-based blocks
+- Correction patterns: `reason_tag` count ≥ 3 in last 4 weeks.
+- Quadrant drift: pivot weeks × quadrants; flag any quadrant < 5% for
+  3+ weeks. Career quadrant gets strongest flag — it's first to collapse.
+- Visibility: weekly count of strategic comments — measurable proxy
+  for the "thin Strategic Leadership" EP3 gap.
+
+### 5. Eagle SOUL.md — corrections_log writeback
+On any user pushback, classify `reason_tag` and `INSERT INTO
+corrections_log` *before* responding. Eagle writes, no separate hook —
+the natural moment to capture a correction is during the conversation.
+
+### 6. Executor v2 — draft-PR flow
+**Single human gate**: Analysis approval. Then autonomous:
+worktree → build → test → branch → push → `gh pr create --draft
+--assignee @me` (no `--reviewer`) → self-review comment → CI loop
+(poll every 10 min, auto-fix up to 3 attempts, escalate on exhaustion).
+
+Hard prohibitions baked into the prompt: never push to `main` /
+`develop` / `release/*`, never non-draft PR, never `--reviewer`, never
+`gh pr merge`, never modify `.github/` or CI configs without explicit
+thread confirmation.
+
+Draft + `assignee=@me` + no reviewers = durable surface visible in
+"assigned to me" without dragging the team into WIP work.
+
+### 7. Microsoft 365 MCP into Hermes
+Wire `outlook_calendar_search` into the Hermes MCP config so
+`inbox-triage`, `weekly-plan`, and `daily-brief` can all check for
+active meetings. STEP 0 gating depends on this. Wire once at the
+config layer, not per prompt.
+
+### 8. Vault consolidation
+Move process docs from `~/Developer/personal-os/` into
+`~/obsidian/work/wiki/`. Runtime files (`system-prompt.md`,
+`agent/prompts/*.md`, SOUL/IDENTITY) **stay in place**.
+**Decision rule**: process/convention → vault; runtime config →
+original location. **Copy** (don't move) `apple-browsers/.cursor/*.mdc`
+files because Cursor IDE still reads them in-tree. Verification step
+is non-negotiable — run each prompt/skill end-to-end after the move.
+
+### 9. Executor PR template (Option B — inline)
+Team PR template requires manual confirmation of task / reviewer /
+description. For the autonomous flow all three are deterministic.
+Inline a variant in the executor prompt that auto-generates the body
+(task link, root cause, fix summary, build/test status, self-review
+checklist, notes for reviewer). Draft state is the safety net.
+
+## Execution Order
+
+Items 1–9 are largely independent; only Executor v2 (item 6 prompt
+rewrite) requires schema (1), vault consolidation (8), and PR template
+(9) first. Each step ends with a concrete test (insert a row, run a
+query, post to thread, etc.).
+
+## "Done" Looks Like
+
+1. Retrospector says specific things ("deferred 4×", "Career <5% for 3
+   weeks", "3 strategic comments vs 0").
+2. `#inbox` stays signal-dense — empty or 1–2 entries most days, no
+   pings during meetings.
+3. `status.md` has a `## ⚡ Career` section.
+4. Pushback in any channel → one DB row, no ceremony.
+5. "fix [task]" in `#executor` → analysis → approval → draft PR within
+   30 min for trivial bugs, CI runs autonomously.
+6. Vault is the one searchable surface for `.mdc`, UI testing skill,
+   and process docs via [[obsidian-mcp]].
+
+## Related
+[[personal-os-state-2026-04-27]] [[personal-os-schema]] [[wiki-ingest-process]] [[ddg-asana-workflow]]

wiki/personal-os-schema.md

@@ -0,0 +1,100 @@
+---
+source: raw/schema.sql
+content_hash: 71b47e47b483834c887de14c76d5b16506d90ad64198d76a92a7b3731132dab8
+namespace: work
+last_synced: 2026-06-01
+confidence: 0.9
+tags: [personal-os, postgres, schema, database]
+---
+
+# Personal OS — Postgres Schema
+
+The `personal_os` database is the durable store behind the Personal OS. It
+holds Asana state, the agent's annotations, the wiki, and semantic memory.
+Created with `psql -U admin -d personal_os -f schema.sql`.
+
+## Extensions
+
+- `vector` — pgvector, used for embeddings (tasks, wiki, memory)
+- `pg_trgm` — trigram GIN index on `tasks.name` for fuzzy search
+
+## Core Tables
+
+### `tasks`
+One row per Asana task GID, upserted every sync. Mirrors enough of the Asana
+task object to answer questions without round-tripping. `source` tracks
+*how* we discovered the task: `my_tasks` | `following` | `delegated` |
+`project` — highest-priority single value when multiple apply. Includes
+`possibly_deleted` + `last_seen_in_full_sync` for soft deletion. The full
+API response is preserved in `raw_json` (JSONB).
+
+### `stories`
+Raw Asana event log per task. `resource_subtype` is Asana's event type
+(`comment_added`, `assigned`, `due_date_changed`, `section_changed`, etc.).
+Both `text` and `html_text` are kept — the HTML lets us parse embedded task
+GID references for [[task-edges]].
+
+### `task_edges`
+Directed graph of task→task relationships. `relation_type`:
+`subtask` | `dependency` | `dependent` | `project_sibling` | `mention`.
+`related_gid` may not yet exist in `tasks` (edges can point to unfetched
+tasks).
+
+### `task_annotations`
+Agent-written (or user-written) notes about tasks. `annotation_type`:
+`irrelevant` | `watching` | `needs_action` | `snoozed`. The `note` column
+holds the agent's reasoning. Tracked by `annotated_by` (`agent` | `user`).
+
+### `sync_state`
+One row per logical sync stream. Keys include `my_tasks`, `following`,
+`delegated`, `project:{gid}`, `workspace_events`. `cursor` is either an ISO
+timestamp (`modified_since`) or an Events API sync_token.
+
+## Embeddings & Wiki
+
+### `task_embeddings`
+1024-dim Voyage AI embeddings per task, HNSW index for cosine similarity.
+
+### `file_references`
+Files / URLs queued for wiki ingest. `bookmark_data` stores macOS
+security-scoped bookmarks (BYTEA). `wiki_stale=true` → next [[wiki-ingest-process]]
+run will process it. `modification_log` (JSONB) keeps a hash-change trail.
+
+### `wiki_pages`
+LLM-synthesised pages. `content` is markdown synthesis, **never** a raw
+copy. `sources` (JSONB) links back to `file_references`. `superseded_by`
+chains old versions. 1536-dim embedding for semantic search.
+`confidence` — 0.6 inferred, 0.8 default, 0.9 structured source, 1.0
+human-written.
+
+### `memory_store`
+Semantic memory accumulated from Discord / Claude / manual entries.
+`type`: `fact` | `preference` | `decision` | `person`. `source`:
+`discord` | `claude` | `manual`. Optional `expires_at` (null = permanent).
+
+## Namespacing
+
+All core tables carry `namespace TEXT NOT NULL DEFAULT 'work'`, constrained
+to `work` | `personal` | `family`. Lets one DB serve multiple Personal OS
+contexts without cross-leakage.
+
+## Views
+
+### `active_tasks`
+Convenience view: not completed, not annotated `irrelevant`, not snoozed
+into the future. Ordered by `source` priority (my_tasks → delegated →
+following → project), then `due_on`, then `modified_at DESC`. The agent's
+default starting point for daily review.
+
+## Key Design Choices
+
+- Asana stays the source of truth — DB is a queryable cache.
+- `raw_json` on tasks preserves un-modeled fields without schema churn.
+- `task_edges.related_gid` is intentionally not a foreign key — we record
+  references to tasks we haven't fetched.
+- Annotations are append-keyed by `(task_gid, annotation_type)` so the
+  agent can have one note per category without overwriting history via
+  the [[corrections-log]] table.
+
+## Related
+[[personal-os-state-2026-04-27]] [[wiki-ingest-process]] [[personal-os-catchup-plan-2026-04-27]] [[ddg-asana-workflow]]

wiki/personal-os-state-2026-04-27.md

@@ -0,0 +1,131 @@
+---
+source: raw/personal-os-state-2026-04-27.md
+content_hash: 1640cb63145b49263b9476e2c39fe443ac561fdf6c27faccc50cfc8568552614
+namespace: work
+last_synced: 2026-06-01
+confidence: 0.8
+tags: [personal-os, state, snapshot, executor]
+---
+
+# Personal OS — State Snapshot (2026-04-27)
+
+Current state after the [[personal-os-catchup-plan-2026-04-27|catch-up
+plan]] executed. Supersedes `state-2026-04-27.md` and
+`personal-os-plan-v3.md`.
+
+## Infrastructure (✅ Live)
+
+| Component | Notes |
+|-----------|-------|
+| PostgreSQL `personal_os` | tasks, stories, task_edges, sync_state, signal_queue, activity_daily — see [[personal-os-schema]] |
+| `sync.js` | 4-source Asana fetch, incremental deltas, `sweepRecentlyCompleted()` |
+| `generate-status.js` | DB → `status.md` + `asana_context.md`, `possibly_deleted` filter fixed |
+| ActivityWatch | DDG browser watcher + Xcode watcher |
+| Hermes v0.11.x | launchd `ai.hermes.gateway`, Claude Code OAuth, `claude-sonnet-4-6` |
+| Discord bot `Орёл#0898` | No `@`-mention required; channels: `#daily-brief #inbox #focus #executor #projects #journal #retrospector` |
+
+## Automation
+
+**Hermes cron** (workdays): data-pipeline `*/30`, morning-brief 08:30,
+eod-summary 18:00, inbox-check `*/30 9-19`, weekly-plan Mon 08:00,
+weekly-review Fri 17:00, retrospector Fri 17:30, commit-vault 23:00.
+
+**launchd** (fires on wake): `personal.os.heartbeat` `:05` hourly
+(watchdog), `personal.os.wiki-ingest` 22:00 daily (see
+[[wiki-ingest-process]]).
+
+## Vault Layout
+
+Git remote: `ssh://truenas_admin@mallexxx.duckdns.org/mnt/RED_2TB/storage/git/obsidian-vault.git`
+
+```
+~/obsidian/
+├── wiki/    ← LLM-generated (this skill) + human-written (confidence 1.0)
+├── raw/     ← symlinks to external sources
+├── work/    ← Eagle write zone (projects/, decisions/, tech-design/)
+├── personal/
+└── family/
+```
+
+## Agent Prompts (`agent/prompts/`)
+`daily-brief`, `weekly-plan`, `weekly-review`, `inbox-triage`,
+`generate-status`, `wiki-ingest`, `executor-bug-fix`,
+`executor-td-review`.
+
+## Catch-up Plan — Execution Status
+
+All ten items shipped:
+1. Schema migration ✅ (corrections_log, executor_runs, activity_daily.quadrant)
+2. MS365 MCP 🟡 — cloud connector works in Claude Code; Hermes `claude -p` will pick it up once exposed to the CLI env. No local npm package available.
+3. `inbox-triage` STEP 0 ✅
+4. Advisor + visibility + executor queue queries ✅ (first run: 5 advisor opps, 8 visibility gaps, 10 executor queue items)
+5. Quadrant classification + `upsertActivityDaily` ✅
+6. Retrospector SQL blocks ✅
+7. SOUL.md writeback to `corrections_log` ✅
+8. Vault consolidation ✅ (43 `.cursor/rules/*.mdc` files in `~/obsidian/work/wiki/apple-browsers/`)
+9. Inline PR template ✅
+10. Executor v2 prompt ✅ (prompt written; not yet exercised on a real task)
+
+## Open Items
+
+| Item | Status |
+|------|--------|
+| Executor v2 end-to-end live run | 🟡 not yet exercised |
+| `work/projects/` | 🟡 empty — Eagle fills on first relevant convo |
+| MS365 MCP in Hermes runtime | 🟡 blocked on CLI env exposure |
+| Phase 2-family | 🔴 deferred until after Phase 4 |
+
+## Executor v2 — Design Summary
+
+On-demand worker that evaluates the backlog, picks a task, spawns a
+sub-agent in a Discord thread under `#executor`, streams progress,
+opens a draft PR, self-reviews, runs a CI poll loop (every 10 min, 3
+auto-fix attempts), and surfaces summaries in daily/weekly briefs.
+
+Trigger modes: user-initiated (`fix [task]` in `#executor`), scheduled
+(Friday weekly-review suggests next), and status-driven
+(`## ⚡ Executor Queue` in `status.md`).
+
+Task qualification SQL: assigned to me, not completed, not
+`possibly_deleted`, no stories in 14 days, due in next 30 days.
+
+Worktree path: `~/DuckDuckGo/apple-browsers.git/.claude/worktrees/executor-{gid}-{slug}/`.
+Scheme: `DuckDuckGo macOS` (workspace required, `xcbeautify` required).
+
+Cleanup on merge/close: `git worktree remove --force` +
+`git push --delete executor/<branch>` + stop/delete VM if used.
+
+## Key Divergences From Plan v3 (Reasoning)
+
+1. **wiki-ingest moved from Hermes cron → launchd `claude -p`** —
+   needs local filesystem (read raw, write vault); Hermes agents only
+   touch the API.
+2. **`personal-wiki` MCP → `obsidian-mcp`** — the custom DB-backed
+   wiki was never finished; standard package over `~/obsidian/`
+   replaces it. `wiki_pages` / `file_references` tables dropped.
+3. **Per-job launchd plists → single hourly heartbeat** — one
+   watchdog reads `jobs.json` and triggers missed one-per-day jobs.
+   Frequent jobs self-recover within Hermes's 2-hour grace window.
+4. **Retrospector split out of weekly-review** — separate prompt,
+   separate cron, separate `#retrospector` channel; weekly-review
+   rewritten without duplicate `OUTPUT FORMAT` blocks.
+5. **NAS git path `/home/...` → `/mnt/RED_2TB/storage/git/`** — boot
+   pool is wiped on TrueNAS OS update; persistent data must live on
+   `/mnt/`.
+6. **Phase 2-family deferred** — VPS + family Hermes + iCloud vault
+   sync waits until Executor v2 (Phase 4) proves itself.
+7. **Executor v1 = manual-only STOP gates** — first autonomous code
+   agent ships conservative; v2 (single gate + PR/CI loop) follows.
+8. **wiki-ingest rewritten DB-first → file-based** — vault is now
+   source of truth; SHA256 content_hash in frontmatter does
+   incremental sync without DB dependency. See [[wiki-ingest-process]].
+
+## Known Tech Debt
+
+- Morning brief not yet smoke-tested in production
+- `executor-bug-fix.md` v1 is the live prompt — needs replacement with v2
+- NAS SSH key still in boot-pool `/home` — must migrate to `/mnt/RED_2TB/`
+- Hermes memory (`~/.hermes/memory/`) not backed up
+
+## Related
+[[personal-os-catchup-plan-2026-04-27]] [[personal-os-schema]] [[wiki-ingest-process]] [[ddg-asana-workflow]]

wiki/wiki-ingest-process.md

@@ -0,0 +1,89 @@
+---
+source: raw/wiki-ingest-prompt.md
+content_hash: 29f6a0914d428cf62eeb2a832d6b3048edf3e1d16aa7714c09f686377394d128
+namespace: work
+last_synced: 2026-06-01
+confidence: 0.9
+tags: [personal-os, wiki, agent-prompt, obsidian]
+---
+
+# Wiki Ingest — Process
+
+Daily process that keeps `~/obsidian/wiki/` synchronised with external
+source files surfaced via symlinks in `~/obsidian/raw/`. Runs as a
+launchd subprocess (`personal.os.wiki-ingest`) invoking `claude -p` —
+not a Hermes cron job, because it needs local filesystem access to read
+sources and write wiki pages.
+
+## Flow
+
+### STEP 0 — Discover changed files
+For each entry in `~/obsidian/raw/`:
+1. Read the symlink target.
+2. Compute `shasum -a 256`.
+3. Scan `~/obsidian/wiki/` for any `.md` whose frontmatter has
+   `source: raw/<filename>`.
+4. If page exists AND `content_hash` matches → **skip** (unchanged).
+5. Otherwise → queue for processing.
+6. If a symlink is broken, skip silently — never create a page for
+   missing content.
+
+Empty queue → print "Wiki is up to date. Nothing to ingest." and exit.
+
+### STEP 1 — Synthesise (per file)
+- Update an existing wiki page if one references the source, otherwise
+  create `~/obsidian/wiki/<topic-name>.md` named by **topic, not source
+  filename** (e.g. `personal-os-schema.md`, not `schema.md`).
+- Write a synthesis: extract key facts, decisions, structure — **never
+  copy verbatim**.
+- Cap at ~800 words; split big topics into linked pages.
+- Use `[[wiki-links]]` for cross-references.
+
+### STEP 2 — Report
+Print processed / created / updated / skipped counts and a list of
+touched pages.
+
+## Frontmatter Schema
+
+```yaml
+source: raw/<filename>
+content_hash: <sha256>
+namespace: work
+last_synced: <YYYY-MM-DD>
+confidence: 0.8
+tags: [tag1, tag2]
+```
+
+`content_hash` and `last_synced` are always updated after a write — they
+are how the next ingest decides what changed.
+
+## Confidence Scale
+
+| Value | Meaning |
+|-------|---------|
+| 0.6 | inferred or partial content |
+| 0.8 | clear single source (default) |
+| 0.9 | highly structured, authoritative source (e.g. SQL schema) |
+| 1.0 | reserved for human-written notes in `work/`, `personal/`, `family/` |
+
+## Hard Rules
+
+- **Never** verbatim-copy source files; always synthesise.
+- **Never** edit files marked `confidence: 1.0` — those are
+  human-authored.
+- **Never** process `namespace: family` files unless explicitly told.
+- Always update `content_hash` and `last_synced` after writing.
+- Broken symlink → skip, do not create.
+
+## Why File-Based, Not DB-Backed
+
+Earlier design (v3) had `wiki_pages` / `file_references` tables drive
+ingest via SQL. That was replaced by file-based ingest because
+[[obsidian-mcp]] now exposes the vault directly — the filesystem is the
+source of truth and the DB tables would have been a duplicate index of
+the same content. The Postgres tables for [[personal-os-schema|wiki_pages]]
+still exist but the agent doesn't read or write them anymore (see
+[[personal-os-state-2026-04-27|Decision 8]]).
+
+## Related
+[[personal-os-schema]] [[personal-os-state-2026-04-27]] [[personal-os-catchup-plan-2026-04-27]]