From 9303eb1bb7d794530345c8d899d8b239891a39a2 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Tue, 28 Apr 2026 01:01:06 +0600 Subject: [PATCH] =?UTF-8?q?2026-04-28:=20vault=20catchup=20=E2=80=94=20pur?= =?UTF-8?q?pose,=20self-modification,=20CI=20pipeline,=20agent=20rules?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add personal-os-purpose.md: ADHD context, design principles, quadrant model - Add personal-os-self-modification.md: Eagle's guide for evolving the system - Update personal-os-agent-rules.md: fix status.md path, corrections_log → SQL, add #retrospector cron, personal project activity notes - Update personal-os-architecture.md: correct paths, add executor/tables/data flow - Add work/wiki/apple-browsers/ci-pipeline.md: full CI schema, xcodebuild commands, local equivalents, VM/executor build commands - Add raw/ symlinks to planning docs (state-2026-04-27-final, catchup-plan-2026-04-27) --- .obsidian/graph.json | 2 +- raw/personal-os-catchup-plan-2026-04-27.md | 1 + raw/personal-os-state-2026-04-27.md | 1 + wiki/personal-os-agent-rules.md | 45 ++++- wiki/personal-os-architecture.md | 155 +++++++++++--- wiki/personal-os-purpose.md | 73 +++++++ wiki/personal-os-self-modification.md | 152 ++++++++++++++ work/wiki/apple-browsers/ci-pipeline.md | 222 +++++++++++++++++++++ 8 files changed, 607 insertions(+), 44 deletions(-) create mode 120000 raw/personal-os-catchup-plan-2026-04-27.md create mode 120000 raw/personal-os-state-2026-04-27.md create mode 100644 wiki/personal-os-purpose.md create mode 100644 wiki/personal-os-self-modification.md create mode 100644 work/wiki/apple-browsers/ci-pipeline.md diff --git a/.obsidian/graph.json b/.obsidian/graph.json index dd437b00..2a7c174d 100644 --- a/.obsidian/graph.json +++ b/.obsidian/graph.json @@ -17,6 +17,6 @@ "repelStrength": 10, "linkStrength": 1, "linkDistance": 250, - "scale": 0.6689230604912264, + "scale": 0.7037000396295, "close": false } \ No newline at end of file diff --git a/raw/personal-os-catchup-plan-2026-04-27.md b/raw/personal-os-catchup-plan-2026-04-27.md new file mode 120000 index 00000000..fe36df4b --- /dev/null +++ b/raw/personal-os-catchup-plan-2026-04-27.md @@ -0,0 +1 @@ +/Users/admin/Developer/personal-os/planning/personal-os-catchup-plan-2026-04-27.md \ No newline at end of file diff --git a/raw/personal-os-state-2026-04-27.md b/raw/personal-os-state-2026-04-27.md new file mode 120000 index 00000000..5cf2d332 --- /dev/null +++ b/raw/personal-os-state-2026-04-27.md @@ -0,0 +1 @@ +/Users/admin/Developer/personal-os/planning/state-2026-04-27-final.md \ No newline at end of file diff --git a/wiki/personal-os-agent-rules.md b/wiki/personal-os-agent-rules.md index f4ea366d..0683c0d9 100644 --- a/wiki/personal-os-agent-rules.md +++ b/wiki/personal-os-agent-rules.md @@ -1,7 +1,7 @@ --- namespace: work tags: [system, agent, eagle, rules] -last_synced: 2026-04-27 +last_updated: 2026-04-28 confidence: 1.0 --- @@ -14,17 +14,17 @@ Eagle (Орёл) is the Discord-facing Hermes agent. This page documents what Ea ``` Asana API → sync.js (every 30 min, workdays via Hermes cron) - → PostgreSQL (tasks, stories, task_edges, sync_state) + → PostgreSQL (tasks, stories, task_edges, sync_state, task_annotations) → generate-status.js - → ~/Developer/personal-os/asana_context.md ← raw Asana data - → ~/Developer/personal-os/status.md ← Eagle reads THIS + → ~/Developer/personal-os/asana_context.md ← raw Asana context (inbox-check) + → ~/context/status.md ← Eagle reads THIS ``` -Eagle **only reads** `status.md`. It never touches the DB directly or calls Asana API. +Eagle **only reads** `~/context/status.md`. It never touches the DB directly or calls Asana API. ## Eagle's Knowledge Sources -1. `~/Developer/personal-os/status.md` — live Asana snapshot (updated every 30 min) +1. `~/context/status.md` — live Asana + ActivityWatch snapshot (updated every 30 min) 2. `~/Developer/personal-os/briefs/daily/YYYY-MM-DD.md` — today's brief 3. `~/Developer/personal-os/briefs/weekly/plan-MONDAY.md` — weekly plan 4. `~/obsidian/wiki/` — knowledge base (via obsidian MCP) @@ -35,13 +35,25 @@ Eagle **only reads** `status.md`. It never touches the DB directly or calls Asan When Alex corrects something: 1. Edit the brief directly (show diff first, confirm before writing) 2. If systemic → edit the prompt template in `agent/prompts/` -3. Log to `~/Developer/personal-os/logs/corrections_log.md`: `YYYY-MM-DD HH:MM | file | what changed` +3. Log correction silently to `corrections_log` table: + +```sql +INSERT INTO corrections_log (date, week_number, source, original_plan, correction, deferred_gids, reason_tag) +VALUES (current_date, EXTRACT(WEEK FROM current_date)::int, + '', '', '', + ARRAY[]::text[], ''); +``` + +`source` values: `discord_pushback` | `morning_brief_correction` | `manual` +`reason_tag` examples: `ship-review-crunch` | `urgent-bug` | `meeting-day` | `over-estimated` | `scope-change` + +Never announce the log insertion. It is silent instrumentation. ## Self-Diagnostics If pipeline seems stale, Eagle can check: ```bash -stat -f "%Sm" ~/Developer/personal-os/status.md +stat -f "%Sm" ~/context/status.md # Should be < 40 min old on workdays ``` @@ -67,9 +79,10 @@ bash ~/scripts/run-wiki-ingest.sh | inbox-check | */30 9-19 workdays | #inbox | | weekly-plan | Mon 08:00 | #daily-brief | | weekly-review | Fri 17:00 | #daily-brief | +| retrospector | Fri 17:30 | #retrospector | | commit-vault | 23:00 daily | silent | -wiki-ingest runs at 22:00 via **launchd** (not Hermes) — because API-based Claude has no Mac filesystem access. +wiki-ingest runs at 22:00 via **launchd** (not Hermes) — because it needs Mac filesystem access. ## Allowed File Writes @@ -79,6 +92,16 @@ Eagle can write to: - `briefs/weekly/YYYY-MM-DD.md` - `briefs/inbox/YYYY-MM-DD.md` - `agent/prompts/*.md` (with confirmation) -- `logs/corrections_log.md` +- `~/obsidian/work/projects/SLUG.md` +- `~/obsidian/work/decisions/YYYY-MM-TOPIC.md` +- `~/obsidian/personal/` +- `~/obsidian/family/` -Never write to `obsidian/wiki/` directly — that's wiki-ingest's job. +Never write to `obsidian/wiki/` directly — that's wiki-ingest's job. +Never write `corrections_log.md` — use the SQL INSERT above (table, not file). + +## Activity Classification + +Work activity is classified via `aw-projects.json`. Projects with `namespace: "personal"` (e.g., AXPressDeck, media_files_db) appear as a footnote in the Activity section of status.md but are **not** counted as work time and **not** upserted to `activity_daily`. + +To re-classify a project (personal ↔ work), ask Eagle to edit `aw-projects.json`. diff --git a/wiki/personal-os-architecture.md b/wiki/personal-os-architecture.md index a3dc7238..9ce4eb9b 100644 --- a/wiki/personal-os-architecture.md +++ b/wiki/personal-os-architecture.md @@ -1,44 +1,135 @@ --- namespace: work tags: [system, architecture] -last_synced: 2026-04-27 +last_updated: 2026-04-28 confidence: 1.0 --- -# Personal OS Architecture +# Personal OS — Architecture -## Стратег (Claude Code, локально) -- `~/scripts/run-pipeline.sh` → sync.js + generate-status.js каждые 30 мин (workdays) -- `~/Developer/personal-os/asana_context.md` — сырые данные Asana -- `~/Developer/personal-os/status.md` — живой срез задач для Орла -- `~/scripts/run-wiki-ingest.sh` → обновляет ~/obsidian/wiki/ (22:00, launchd) +## Why It Exists -## Орёл (Hermes v0.11.x, Discord бот) -- Читает только `~/Developer/personal-os/status.md` — никогда не идёт в БД напрямую -- Discord каналы: #daily-brief #inbox #focus #executor #projects #journal -- Ищет знания через obsidian MCP по vault -- Логирует корректировки в `~/Developer/personal-os/logs/corrections_log.md` -- Cron jobs: morning-brief (08:30), eod-summary (18:00), inbox-check (30 мин), weekly-plan (Пн 08:00), weekly-review (Пт 17:00) +See: `personal-os-purpose.md` -## Vault (~/obsidian/) -- `wiki/` — LLM-синтез, генерируется wiki-ingest, не редактировать руками -- `raw/` — symlinks на внешние файлы проекта -- `work/`, `personal/`, `family/` — заметки, редактировать свободно -- Git репозиторий, автокоммит в 23:00 с агентным summary +## Agents -## База данных (Postgres, только Asana + activity) -- tasks, stories, task_edges, sync_state -- signal_queue, activity_daily -- Нет wiki данных — всё в vault файлах +### Стратег (Strategist) — data pipeline, no conversation +- `~/scripts/run-pipeline.sh` → sync.js + generate-status.js every 30 min (workdays via Hermes cron) +- Writes `~/context/status.md` and `~/Developer/personal-os/asana_context.md` +- Writes daily AW activity summaries to `activity_daily` table with quadrant classification +- No Discord output — pure data layer -## Почему wiki-ingest через launchd, не Hermes -Claude через API не имеет доступа к файловой системе Mac. -wiki-ingest пишет файлы в ~/obsidian/wiki/ — это требует -`claude -p` запущенного локально, не API вызова. +### Орёл / Eagle (Tactician + reactive layer) — Hermes v0.11.x Discord bot +- Reads `~/context/status.md`, daily/weekly briefs, vault via obsidian MCP +- Discord channels: `#daily-brief` `#inbox` `#focus` `#executor` `#projects` `#journal` `#retrospector` +- Cron jobs: morning-brief (08:30 MTWRF), eod-summary (18:00 MTWRF), inbox-check (30 min 09–19 MTWRF), weekly-plan (Mon 08:00), weekly-review (Fri 17:00), retrospector (Fri 17:30), commit-vault (23:00 daily) +- Logs corrections to `corrections_log` table (SQL INSERT, silently on pushback) +- **Never accesses Asana API directly** — only reads pre-rendered markdown and DB -## Файлы конфигурации -- `~/.hermes/config.yaml` — Hermes config, модель, Discord, cron -- `~/.hermes/SOUL.md` — характер и правила Орла -- `~/.hermes/auth.json` — Claude Code OAuth (credential_pool) -- `~/Developer/personal-os/config.json` — Asana workspace/user GIDs -- `~/.config/personal-os/env` — env vars (ASANA_API_KEY, POSTGRES_URL) +### Ретроспектор (Retrospector) — Friday pattern analysis +- Runs as separate Hermes cron at 17:30 Friday → `#retrospector` +- Sources: corrections_log table, status.md subtask counts, obsidian git log, activity_daily quadrants, stories visibility metric +- Prompt: `~/Developer/personal-os/agent/prompts/retrospector.md` + +### Исполнитель (Executor) — on-demand autonomous worker +- Triggered by explicit "fix bug [GID]" in #executor — never autonomous +- Creates git worktree at `~/DuckDuckGo/apple-browsers.git/.claude/worktrees/executor-{gid}-{slug}/` +- Single STOP gate after analysis; then autonomous: fix → build → test → draft PR → CI loop +- State machine in `executor_runs` table +- Prompt: `~/Developer/personal-os/agent/prompts/executor-bug-fix.md` +- For UI tests: spins VM via ddg-vm MCP (virfield), cleans up on PR close/merge + +## Data Flow + +``` +Asana API + → sync.js (30 min, Hermes cron, silent) + → PostgreSQL: tasks, stories, task_edges, sync_state, task_annotations + +ActivityWatch (local daemon) + → generate-status.js (reads AW HTTP API) + → activity_daily (upserts daily project/quadrant summaries) + +generate-status.js + → ~/context/status.md ← Eagle reads this + → ~/Developer/personal-os/asana_context.md ← inbox-check reads this + +Eagle corrections in Discord + → corrections_log table (INSERT on pushback, no announcement) + → Retrospector reads weekly + +Executor runs + → executor_runs table (state machine) + → ~/DuckDuckGo/apple-browsers.git/.claude/worktrees/ + → GitHub draft PRs (--draft, --assignee @me, no reviewers) +``` + +## Key File Locations + +| File / Path | Purpose | +|-------------|---------| +| `~/context/status.md` | Live Asana + AW snapshot. Written by generate-status.js. Eagle reads this. | +| `~/Developer/personal-os/asana_context.md` | Raw Asana context for inbox-check | +| `~/Developer/personal-os/agent/prompts/` | All agent prompts (Hermes cron + on-demand) | +| `~/.hermes/SOUL.md` | Eagle's identity, rules, vault write permissions | +| `~/Developer/personal-os/config.json` | Asana workspace/user GIDs, section GIDs | +| `~/Developer/personal-os/aw-projects.json` | ActivityWatch project classification rules | +| `~/.hermes/config.yaml` | Hermes config: model, MCP servers, cron settings | +| `~/obsidian/` | Vault — git repo, NAS remote, obsidian-mcp for search | +| `~/Developer/personal-os/logs/` | heartbeat.log, wiki-ingest-YYYY-MM-DD.md | +| `~/Developer/personal-os/briefs/` | daily/, weekly/, inbox/ — agent-written brief files | +| `~/DuckDuckGo/apple-browsers.git/` | Browser repo (bare), worktrees as sibling dirs | +| `~/DuckDuckGo/apple-browsers.git/.claude/worktrees/` | Executor worktrees | + +## Database Tables + +| Table | Written by | Read by | +|-------|-----------|---------| +| tasks | sync.js | generate-status.js, executor, status queries | +| stories | sync.js | generate-status.js, executor, retrospector | +| task_edges | sync.js | subtask traversal | +| sync_state | sync.js | generate-status.js header | +| task_annotations | manual / Eagle | generate-status.js irrelevant filter | +| activity_daily | generate-status.js | retrospector quadrant drift | +| corrections_log | Eagle on pushback | retrospector pattern analysis | +| executor_runs | executor | generate-status.js active runs, executor state machine | + +## MCP Servers + +**Hermes agents:** +| Name | Command | Used for | +|------|---------|---------| +| obsidian | `/opt/homebrew/bin/obsidian-mcp ~/obsidian` | Vault search, read, write | + +**Claude Code / `claude -p` sessions:** +| Name | Command | Used for | +|------|---------|---------| +| ddg-vm | `npx tsx ~/Developer/virfield/server/mcp-server.ts` | VM lifecycle for UI tests | +| obsidian | `/opt/homebrew/bin/obsidian-mcp ~/obsidian` | Vault access | + +## launchd Agents + +| Label | Schedule | Purpose | +|-------|----------|---------| +| personal.os.heartbeat | :05 every hour | Watchdog — triggers missed Hermes jobs after wake | +| personal.os.wiki-ingest | 22:00 daily | `claude -p` wiki synthesis (needs filesystem access) | + +## Vault Structure + +``` +~/obsidian/ +├── wiki/ ← LLM-generated + hand-written (confidence: 1.0 = never overwrite) +│ ├── personal-os-architecture.md (this file) +│ ├── personal-os-purpose.md +│ ├── personal-os-agent-rules.md +│ ├── personal-os-self-modification.md +│ ├── personal-os-sync-pipeline.md +│ ├── personal-os-schema.md (auto-generated) +│ ├── ddg-asana-workflow.md +│ └── vault-filling-guide.md +├── raw/ ← symlinks to external source files (schema.sql, etc.) +├── work/ +│ └── wiki/ +│ └── apple-browsers/ ← .cursor/rules copies (.md) + vm-ui-testing.md +└── personal/ family/ +``` diff --git a/wiki/personal-os-purpose.md b/wiki/personal-os-purpose.md new file mode 100644 index 00000000..93e43055 --- /dev/null +++ b/wiki/personal-os-purpose.md @@ -0,0 +1,73 @@ +--- +namespace: work +tags: [system, purpose, adhd, design] +last_updated: 2026-04-28 +confidence: 1.0 +--- + +# Personal OS — Purpose + +## Why It Exists + +Alex is a macOS Browser Developer / DRI at DuckDuckGo. Async-first company, primary channels Asana and Slack. ADHD profile. + +The Personal OS exists to hold work context **outside Alex's head** — so that context isn't lost during deep focus, transitions, or when switching back from a meeting. + +Without it, three things happen regularly: +1. A comment sits on an Asana task for days — Alex never saw it because he was in Xcode +2. A career-level task (Ship Review, Tech Design review) gets deferred until it becomes a bottleneck +3. End of day: "what did I actually do today?" — no clear answer, no signal for tomorrow + +## Design Principles + +**Principle 1 — Context is the product, not notifications.** +The system's job is to maintain a live, accurate picture of what's happening. Eagle reads that picture and speaks when something actually matters. It is not a notification firehose. + +**Principle 2 — Protect focus.** +If Alex is in Xcode or Cursor, don't interrupt unless there's a red signal older than 3 hours. ADHD makes context switches expensive — the system should absorb noise, not amplify it. + +**Principle 3 — No hallucinated state.** +Eagle never guesses about Asana tasks. All Asana data flows through sync.js → PostgreSQL → generate-status.js → status.md. Eagle reads markdown, not the API. This means the data is always consistent and never stale by more than 30 minutes. + +**Principle 4 — Career goals are first-class.** +The system explicitly tracks: Project Advisor opportunities, visibility gaps (Ship Reviews where Alex hasn't commented), Tech Design reviews pending. These surface in status.md and the daily brief. Without explicit tracking they'd be invisible. + +**Principle 5 — No new infra.** +The system has enough moving parts. New capabilities go into existing components: Postgres tables, generate-status.js queries, prompt files in `agent/prompts/`, SOUL.md rules. No new YAML files, no new shell scripts, no new launchd plists unless there is no other way. + +## The Four Agents + +| Agent | Role | When | +|-------|------|------| +| **Стратег (Strategist)** | Data pipeline — reads Asana + ActivityWatch, writes status.md | Every 30 min, workdays | +| **Орёл / Eagle (Tactician)** | Discord-facing reactive agent — briefs, inbox, focus gating | Cron + on-demand | +| **Ретроспектор (Retrospector)** | Weekly pattern analysis — corrections, quadrant drift, visibility | Friday 17:30 | +| **Исполнитель (Executor)** | On-demand bug-fix worker — analysis → fix → PR → CI loop | Explicit trigger only | + +Eagle is the only agent Alex directly talks to. The others are background infrastructure. + +## What "Work Context" Means + +The system tracks four categories, mapped to `activity_daily.quadrant`: + +| Quadrant | What it covers | Why it matters | +|----------|---------------|----------------| +| `project` | The current top-priority feature/bug | Time here should dominate most weeks | +| `aor` | AOR maintenance: code review, bug fixes, blocklist | Required to keep the area healthy | +| `career` | Project Advisor, Ship Reviews, Tech Design, visibility | Easily deferred; system forces it into view | +| `strategy` | Planning, Personal OS, Asana organisation | Meta-work; needs a floor, not a ceiling | + +Personal pet projects (AXPressDeck, media_files_db, etc.) are tracked separately and not counted as work time. See `aw-projects.json` (`namespace: "personal"` entries). + +## What It Does Not Do + +- Does not make decisions for Alex — it informs +- Does not send notifications during focus (deep focus gate in inbox STEP 0) +- Does not access Asana API directly from Eagle — only reads pre-rendered status.md +- Does not create PRs, commits, or messages without explicit confirmation (except Executor, after its single approval gate) + +## See Also + +- `personal-os-architecture.md` — full technical architecture, agents, data flow, file locations +- `personal-os-agent-rules.md` — Eagle's operating rules and allowed writes +- `personal-os-self-modification.md` — how Eagle (and Alex) can evolve the system diff --git a/wiki/personal-os-self-modification.md b/wiki/personal-os-self-modification.md new file mode 100644 index 00000000..c299b1b3 --- /dev/null +++ b/wiki/personal-os-self-modification.md @@ -0,0 +1,152 @@ +--- +namespace: work +tags: [system, self-modification, eagle, meta] +last_updated: 2026-04-28 +confidence: 1.0 +--- + +# Personal OS — Self-Modification Guide + +How Eagle (and Alex via Claude Code) can safely evolve the system without breaking it. + +**Rule**: Always show a diff/draft first, confirm, then apply. Commit the vault after. + +--- + +## What Eagle Can Change (no approval needed — show draft first) + +### 1. Prompt files (`~/Developer/personal-os/agent/prompts/*.md`) + +Eagle reads and writes its own prompts. When Alex says "add X to the morning brief" or "change how inbox triage handles Y": + +1. Read the current prompt: `~/Developer/personal-os/agent/prompts/.md` +2. Draft the change, show the diff in Discord +3. On confirmation: write the file +4. Commit: `cd ~/Developer/personal-os && git add agent/prompts/.md && git commit -m "[YYYY-MM-DD] prompt: "` + +**Do not** change `executor-bug-fix.md` without explicit confirmation — it controls autonomous code changes. + +### 2. SOUL.md (`~/.hermes/SOUL.md`) + +Eagle's identity, rules, and operating context. Eagle can propose changes to: +- Allowed vault write paths +- Focus gating rules +- Channel routing + +Same flow: draft → confirm → write → no vault commit needed (SOUL.md is outside vault). + +### 3. ActivityWatch project rules (`~/Developer/personal-os/aw-projects.json`) + +To classify a new project as work or personal: +1. Read current `aw-projects.json` +2. Add entry with `project`, `namespace` (omit for work, `"personal"` for pet projects), and `rules` +3. Show draft, confirm, write +4. Commit: `cd ~/Developer/personal-os && git add aw-projects.json && git commit -m "[YYYY-MM-DD] aw: classify "` + +The `namespace: "personal"` field excludes a project from work activity in status.md and activity_daily. + +### 4. Vault knowledge pages (`~/obsidian/wiki/`, `~/obsidian/work/wiki/`) + +Eagle does **not** write to `wiki/` — that's the wiki-ingest job (22:00 launchd). But Eagle **can** write to: +- `~/obsidian/work/projects/SLUG.md` — project notes +- `~/obsidian/work/decisions/YYYY-MM-TOPIC.md` — decisions +- `~/obsidian/personal/` — personal notes +- `~/obsidian/family/` — family docs + +After writing: `cd ~/obsidian && git add -A && git commit -m "[YYYY-MM-DD] "` + +### 5. Correction logging (corrections_log table) + +Eagle silently logs a row whenever Alex pushes back on a suggestion. No approval needed — this is continuous background instrumentation, not a visible change. + +```sql +INSERT INTO corrections_log (date, week_number, source, original_plan, correction, deferred_gids, reason_tag) +VALUES (current_date, EXTRACT(WEEK FROM current_date)::int, + '', '', '', + ARRAY[]::text[], ''); +``` + +--- + +## What Requires Approval (always confirm explicitly) + +### Schema changes (PostgreSQL) + +New tables, columns, or indexes require a SQL migration. Pattern: +1. Draft the SQL in Discord, explain the purpose +2. On "go": run via `psql personal_os -c "..."` +3. Update `~/obsidian/wiki/personal-os-schema.md` (this is auto-generated at 22:00, but a manual update is fine) + +Current tables: tasks, stories, task_edges, sync_state, task_annotations, activity_daily, corrections_log, executor_runs. + +### New Hermes cron jobs + +Add to `~/.hermes/config.yaml`. Format: +```yaml +crons: + - id: my-new-job + schedule: "30 9 * * 1-5" + prompt: | + Run: ~/Developer/personal-os/agent/prompts/my-prompt.md + channel: "#channel-name" +``` +Must confirm before writing — a broken cron syntax silently prevents Hermes from starting. + +### New launchd agents (`~/Library/LaunchAgents/`) + +New plist files. Rarely needed — only if a job requires Mac filesystem access and can't run inside Hermes (e.g., wiki-ingest). Confirm before writing and before loading with `launchctl`. + +### Executor prompt (`executor-bug-fix.md`) + +Controls autonomous code changes. Changes here need explicit approval because a mistake could cause the Executor to behave incorrectly on real PRs. + +### Hermes model or MCP configuration (`~/.hermes/config.yaml`) + +Model upgrades, new MCP servers. Show the diff; confirm before writing. + +--- + +## How generate-status.js Gets Extended + +When a new data signal should appear in status.md: + +1. Add a SQL query function (e.g., `getNewSignal()`) +2. Add a render function (e.g., `renderNewSignal(data)`) +3. Add both to `main()`: Promise.all for the query, the render call in `parts` +4. Test: `node ~/Developer/personal-os/generate-status.js` and read `~/context/status.md` +5. Commit + +**Never** edit generate-status.js to change how Asana sync works — that's sync.js territory. + +--- + +## Debugging the System + +| Symptom | Check | +|---------|-------| +| status.md is stale (> 40 min old) | `stat -f "%Sm" ~/context/status.md` — if old, run `bash ~/scripts/run-pipeline.sh` | +| Hermes cron didn't fire | Check heartbeat: `tail -20 ~/Developer/personal-os/logs/heartbeat.log` | +| Eagle gave wrong Asana data | The DB may be stale — run `node ~/Developer/personal-os/sync.js` manually | +| activity_daily missing today | generate-status.js runs upsert — check `select * from activity_daily where bucket_day = current_date` | +| corrections_log empty | Eagle only inserts on pushback — expected to be sparse | +| Executor in stuck state | `SELECT id, state, updated_at FROM executor_runs WHERE state NOT IN ('complete','abandoned');` | + +--- + +## Safe Operations Checklist + +Before making any system change: +- [ ] Read the current file/config before proposing a change +- [ ] Show the diff, not just a description +- [ ] Wait for explicit confirmation +- [ ] Apply the change +- [ ] Verify: run the component or check output +- [ ] Commit if in a git repo (`personal-os` or `obsidian`) + +--- + +## See Also + +- `personal-os-architecture.md` — full system map +- `personal-os-agent-rules.md` — Eagle's rules and channel routing +- `personal-os-purpose.md` — why this system exists diff --git a/work/wiki/apple-browsers/ci-pipeline.md b/work/wiki/apple-browsers/ci-pipeline.md new file mode 100644 index 00000000..e1821aab --- /dev/null +++ b/work/wiki/apple-browsers/ci-pipeline.md @@ -0,0 +1,222 @@ +--- +source: ~/DuckDuckGo/apple-browsers.git/main/.github/workflows/ +confidence: 1.0 +namespace: work +last_updated: 2026-04-28 +tags: [ci, github-actions, xcodebuild, testing, build] +--- + +# CI Pipeline — macOS Browser + +Reference for the GH Actions CI setup and how to run equivalent checks locally. + +**Xcode version**: `26.4` (from `.xcode-version`) +**Working directory for all macOS jobs**: `macOS/` inside the repo + +--- + +## Workflows Overview + +| Workflow file | Trigger | What it does | +|---------------|---------|--------------| +| `macos_pr_checks.yml` | PR + push to main/release/hotfix | Unit tests, integration tests, release build, translations | +| `macos_ui_tests.yml` | PR (BSK changes) + push + schedule (3AM UTC) | Full UITest suite across macOS 14/15/26 | +| `macos_performance_tests.yml` | PR + push + daily 5AM UTC | Performance benchmarks (notarized build required) | +| `macos_pir_end_to_end_tests.yml` | PR + push + schedule | PIR feature E2E tests | +| `macos_build_notarized.yml` | Called by other workflows | Builds notarized Review .app + optional DMG | +| `macos_release.yml` | Manual + release process | Full release build and publish | +| `macos_check_sparkle_update.yml` | Post-release | Validates Sparkle update feed | + +--- + +## PR Checks (`macos_pr_checks.yml`) + +Runs on every PR. Three parallel tracks: + +### Track 1: Unit + Integration Tests + +Two matrix flavors run simultaneously: + +| Flavor | Scheme | Runner | active-arch | +|--------|--------|--------|-------------| +| Non-Sandbox | `macOS Browser` | `macos-26` | YES | +| Sandbox | `macOS Browser App Store` | `macos-26-xlarge` | NO | + +**Unit tests command:** +```bash +cd macOS +set -o pipefail && xcodebuild test \ + -scheme "macOS Browser" \ + -derivedDataPath "DerivedData" \ + -configuration "CI" \ + -skipPackagePluginValidation -skipMacroValidation \ + -test-timeouts-enabled YES \ + -default-test-execution-time-allowance 60 \ + -maximum-test-execution-time-allowance 120 \ + ENABLE_TESTABILITY=true \ + ONLY_ACTIVE_ARCH=YES \ + COMPILATION_CACHE_ENABLE_CACHING=YES \ + "-skip-testing:Integration Tests" \ + 2>&1 | xcbeautify +``` + +**Integration tests command** (same flags, opposite -testing: flag): +```bash + "-only-testing:Integration Tests" \ + -retry-tests-on-failure \ +``` + +### Track 2: Release Build + +```bash +cd macOS +set -o pipefail && xcodebuild \ + -scheme "macOS Browser" \ + -derivedDataPath "DerivedData" \ + -configuration "Release" \ + -skipPackagePluginValidation -skipMacroValidation \ + COMPILATION_CACHE_ENABLE_CACHING=YES \ + 2>&1 | xcbeautify +``` + +### Track 3: ShellCheck + Bats (shell scripts) + +```bash +# Bats shell tests +brew install bats-core +cd macOS && bats --formatter junit scripts/tests/* +``` + +--- + +## UI Tests (`macos_ui_tests.yml`) + +### How it works in CI + +1. **Build notarized app first** (`macos_build_notarized.yml`) — runs as a dependency +2. **Setup**: lists all `UITestCase`-derived classes from `UITests/` directory +3. **Matrix**: each test class × each macOS version runs as a separate job +4. **Retry**: `-test-iterations 2 -retry-tests-on-failure -test-repetition-relaunch-enabled YES` + +### Running locally (two approaches) + +**Option A — Direct xcodebuild (host only, no VM):** +```bash +cd ~/DuckDuckGo/apple-browsers.git/main/macOS + +# Build for testing first +set -o pipefail && xcodebuild build-for-testing \ + -scheme "macOS UI Tests CI" \ + -derivedDataPath DerivedData \ + -skipPackagePluginValidation -skipMacroValidation \ + 2>&1 | xcbeautify + +# Run a specific test class +defaults write com.duckduckgo.macos.browser.review moveToApplicationsFolderAlertSuppress 1 +set -o pipefail && xcodebuild test-without-building \ + -scheme "macOS UI Tests CI" \ + -derivedDataPath DerivedData \ + -skipPackagePluginValidation -skipMacroValidation \ + '-only-testing:UI Tests/AutocompleteTests' \ + 2>&1 | xcbeautify +``` + +**Option B — Via VM (virfield/ddg-vm MCP, matches CI environment):** +``` +# 1. Build on host first (DerivedData must exist) +xcodebuild build-for-testing -scheme "macOS UI Tests CI" -derivedDataPath DerivedData ... + +# 2. Start VM session +vm_prepare_session() → { vmName, ip } + +# 3. Run tests in VM +run_tests(vm_id: vmName, scheme: "DuckDuckGo macOS", workspace: "") + +# 4. Poll +get_test_run_status(vm_id: vmName, run_id: run_id) +get_test_results(vm_id: vmName, run_id: run_id) + +# 5. On failure: inspect +peekaboo_image(vm_id: vmName) +get_crash_reports(vm_id: vmName) + +# 6. Always clean up +vm_stop(vm_id: vmName) +vm_delete(vm_id: vmName) +``` + +See `vm-ui-testing.md` for full ddg-vm MCP reference. + +### Key notes + +- All UI test classes **must** inherit from `UITestCase`, not `XCTestCase` — CI enforces this +- CI uses a notarized **Review** build (`DuckDuckGo Review.app`), not Debug +- Screen resolution is set to 1920×1080 in CI (macOS 15+) — VM does the same +- Build on host before running in VM — `run_tests` uses the pre-built xctestrun from `DerivedData` + +--- + +## Executor Build Command (for Executor agent) + +The Executor uses Debug configuration without code signing or notarization: + +```bash +cd ~/DuckDuckGo/apple-browsers.git/.claude/worktrees/executor-{gid}-{slug}/macOS +set -e -o pipefail && xcodebuild \ + -workspace DuckDuckGo-macOS.xcworkspace \ + -scheme "DuckDuckGo macOS" \ + -configuration Debug \ + build \ + 2>&1 | xcbeautify +``` + +Key differences from CI: +- **No** `-derivedDataPath` override — uses default `DerivedData` in worktree +- **No** `ONLY_ACTIVE_ARCH` — defaults to YES for Debug, faster +- **No** `COMPILATION_CACHE_ENABLE_CACHING=YES` — not needed for single-run +- **No** `-skipPackagePluginValidation` unless you hit plugin errors (add if needed) + +--- + +## Xcode Scheme Reference + +| Scheme | Purpose | Used for | +|--------|---------|---------| +| `macOS Browser` | Non-Sandbox (direct distribution) | Unit tests, release build | +| `macOS Browser App Store` | Sandbox variant | Unit tests (sandbox) | +| `macOS UI Tests CI` | UI test runner | UI tests (requires notarized Review app) | +| `DuckDuckGo macOS` | Development scheme | Executor Debug build | + +--- + +## CI Runner Reference + +| Runner | Used for | +|--------|---------| +| `macos-26` | Unit tests (Non-Sandbox), release build | +| `macos-26-xlarge` | Unit tests (Sandbox), UI tests (latest macOS) | +| `macos-15-xlarge` | UI tests on macOS 15 | +| `macos-14-xlarge` | UI tests on macOS 14 | +| `ubuntu-latest` | ShellCheck, translation checks, Asana tasks | + +--- + +## Common Local Equivalents + +| CI step | Local command | +|---------|--------------| +| Select Xcode version | `sudo xcode-select -s /Applications/Xcode_26.4.app` (check `.xcode-version`) | +| Sync code signing | `cd macOS && bundle exec fastlane sync_signing_ci` | +| Build + unit test | See "Unit tests command" above | +| Build for UI test | `xcodebuild build-for-testing -scheme "macOS UI Tests CI" ...` | +| Run one UI test class | `xcodebuild test-without-building ... '-only-testing:UI Tests/ClassName'` | +| Check private API usage | `cd macOS && ./scripts/find_private_symbols.sh "DerivedData/Build/Products/..."` | +| Verify npm bundles | `npm run rebuild-autoconsent --workspace=macOS` | + +--- + +## Cross-Reference + +- `vm-ui-testing.md` — full virfield/ddg-vm MCP reference for VM-based UI tests +- `testing.mdc` → `testing.md` — UITestCase patterns, feature flag setup in tests +- `development-commands.mdc` → `development-commands.md` — team-approved xcodebuild flags