git_admin/obsidian-vault
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

2026-04-28: vault catchup — purpose, self-modification, CI pipeline, agent rules

Browse Source 
- Add personal-os-purpose.md: ADHD context, design principles, quadrant model
- Add personal-os-self-modification.md: Eagle's guide for evolving the system
- Update personal-os-agent-rules.md: fix status.md path, corrections_log → SQL, add #retrospector cron, personal project activity notes
- Update personal-os-architecture.md: correct paths, add executor/tables/data flow
- Add work/wiki/apple-browsers/ci-pipeline.md: full CI schema, xcodebuild commands, local equivalents, VM/executor build commands
- Add raw/ symlinks to planning docs (state-2026-04-27-final, catchup-plan-2026-04-27)
This commit is contained in:
Alexey Martemyanov
2026-04-28 01:01:06 +06:00
parent f280819df4
commit 9303eb1bb7
8 changed files with 607 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.obsidian/graph.json
Vendored
+1 -1
View File
@@ -17,6 +17,6 @@
"repelStrength": 10,
"linkStrength": 1,
"linkDistance": 250,
"scale": 0.6689230604912264,
"scale": 0.7037000396295,
"close": false
}
raw/personal-os-catchup-plan-2026-04-27.md
Symlink
+1
View File
@@ -0,0 +1 @@
/Users/admin/Developer/personal-os/planning/personal-os-catchup-plan-2026-04-27.md
raw/personal-os-state-2026-04-27.md
Symlink
+1
View File
@@ -0,0 +1 @@
/Users/admin/Developer/personal-os/planning/state-2026-04-27-final.md
wiki/personal-os-agent-rules.md
+34 -11
View File
@@ -1,7 +1,7 @@
---
namespace: work
tags: [system, agent, eagle, rules]
last_synced: 2026-04-27
last_updated: 2026-04-28
confidence: 1.0
---
@@ -14,17 +14,17 @@ Eagle (Орёл) is the Discord-facing Hermes agent. This page documents what Ea
```
Asana API
→ sync.js (every 30 min, workdays via Hermes cron)
→ PostgreSQL (tasks, stories, task_edges, sync_state)
→ PostgreSQL (tasks, stories, task_edges, sync_state, task_annotations)
→ generate-status.js
→ ~/Developer/personal-os/asana_context.md ← raw Asana data
→ ~/Developer/personal-os/status.md ← Eagle reads THIS
→ ~/Developer/personal-os/asana_context.md ← raw Asana context (inbox-check)
→ ~/context/status.md ← Eagle reads THIS
```
Eagle **only reads** `status.md`. It never touches the DB directly or calls Asana API.
Eagle **only reads** `~/context/status.md`. It never touches the DB directly or calls Asana API.
## Eagle's Knowledge Sources
1. `~/Developer/personal-os/status.md` — live Asana snapshot (updated every 30 min)
1. `~/context/status.md` — live Asana + ActivityWatch snapshot (updated every 30 min)
2. `~/Developer/personal-os/briefs/daily/YYYY-MM-DD.md` — today's brief
3. `~/Developer/personal-os/briefs/weekly/plan-MONDAY.md` — weekly plan
4. `~/obsidian/wiki/` — knowledge base (via obsidian MCP)
@@ -35,13 +35,25 @@ Eagle **only reads** `status.md`. It never touches the DB directly or calls Asan
When Alex corrects something:
1. Edit the brief directly (show diff first, confirm before writing)
2. If systemic → edit the prompt template in `agent/prompts/`
3. Log to `~/Developer/personal-os/logs/corrections_log.md`: `YYYY-MM-DD HH:MM | file | what changed`
3. Log correction silently to `corrections_log` table:
```sql
INSERT INTO corrections_log (date, week_number, source, original_plan, correction, deferred_gids, reason_tag)
VALUES (current_date, EXTRACT(WEEK FROM current_date)::int,
'<source>', '<original>', '<user_message>',
ARRAY[<gids>]::text[], '<reason_tag>');
```
`source` values: `discord_pushback` | `morning_brief_correction` | `manual`
`reason_tag` examples: `ship-review-crunch` | `urgent-bug` | `meeting-day` | `over-estimated` | `scope-change`
Never announce the log insertion. It is silent instrumentation.
## Self-Diagnostics
If pipeline seems stale, Eagle can check:
```bash
stat -f "%Sm" ~/Developer/personal-os/status.md
stat -f "%Sm" ~/context/status.md
# Should be < 40 min old on workdays
```
@@ -67,9 +79,10 @@ bash ~/scripts/run-wiki-ingest.sh
| inbox-check | */30 9-19 workdays | #inbox |
| weekly-plan | Mon 08:00 | #daily-brief |
| weekly-review | Fri 17:00 | #daily-brief |
| retrospector | Fri 17:30 | #retrospector |
| commit-vault | 23:00 daily | silent |
wiki-ingest runs at 22:00 via **launchd** (not Hermes) — because API-based Claude has no Mac filesystem access.
wiki-ingest runs at 22:00 via **launchd** (not Hermes) — because it needs Mac filesystem access.
## Allowed File Writes
@@ -79,6 +92,16 @@ Eagle can write to:
- `briefs/weekly/YYYY-MM-DD.md`
- `briefs/inbox/YYYY-MM-DD.md`
- `agent/prompts/*.md` (with confirmation)
- `logs/corrections_log.md`
- `~/obsidian/work/projects/SLUG.md`
- `~/obsidian/work/decisions/YYYY-MM-TOPIC.md`
- `~/obsidian/personal/`
- `~/obsidian/family/`
Never write to `obsidian/wiki/` directly — that's wiki-ingest's job.
Never write to `obsidian/wiki/` directly — that's wiki-ingest's job.
Never write `corrections_log.md` — use the SQL INSERT above (table, not file).
## Activity Classification
Work activity is classified via `aw-projects.json`. Projects with `namespace: "personal"` (e.g., AXPressDeck, media_files_db) appear as a footnote in the Activity section of status.md but are **not** counted as work time and **not** upserted to `activity_daily`.
To re-classify a project (personal ↔ work), ask Eagle to edit `aw-projects.json`.
wiki/personal-os-architecture.md
+123 -32
View File
@@ -1,44 +1,135 @@
---
namespace: work
tags: [system, architecture]
last_synced: 2026-04-27
last_updated: 2026-04-28
confidence: 1.0
---
# Personal OS Architecture
# Personal OS — Architecture
## Стратег (Claude Code, локально)
- `~/scripts/run-pipeline.sh` → sync.js + generate-status.js каждые 30 мин (workdays)
- `~/Developer/personal-os/asana_context.md` — сырые данные Asana
- `~/Developer/personal-os/status.md` — живой срез задач для Орла
- `~/scripts/run-wiki-ingest.sh` → обновляет ~/obsidian/wiki/ (22:00, launchd)
## Why It Exists
## Орёл (Hermes v0.11.x, Discord бот)
- Читает только `~/Developer/personal-os/status.md` — никогда не идёт в БД напрямую
- Discord каналы: #daily-brief #inbox #focus #executor #projects #journal
- Ищет знания через obsidian MCP по vault
- Логирует корректировки в `~/Developer/personal-os/logs/corrections_log.md`
- Cron jobs: morning-brief (08:30), eod-summary (18:00), inbox-check (30 мин), weekly-plan (Пн 08:00), weekly-review (Пт 17:00)
See: `personal-os-purpose.md`
## Vault (~/obsidian/)
- `wiki/` — LLM-синтез, генерируется wiki-ingest, не редактировать руками
- `raw/` — symlinks на внешние файлы проекта
- `work/`, `personal/`, `family/` — заметки, редактировать свободно
- Git репозиторий, автокоммит в 23:00 с агентным summary
## Agents
## База данных (Postgres, только Asana + activity)
- tasks, stories, task_edges, sync_state
- signal_queue, activity_daily
- Нет wiki данных — всё в vault файлах
### Стратег (Strategist) — data pipeline, no conversation
- `~/scripts/run-pipeline.sh` → sync.js + generate-status.js every 30 min (workdays via Hermes cron)
- Writes `~/context/status.md` and `~/Developer/personal-os/asana_context.md`
- Writes daily AW activity summaries to `activity_daily` table with quadrant classification
- No Discord output — pure data layer
## Почему wiki-ingest через launchd, не Hermes
Claude через API не имеет доступа к файловой системе Mac.
wiki-ingest пишет файлы в ~/obsidian/wiki/ — это требует
`claude -p` запущенного локально, не API вызова.
### Орёл / Eagle (Tactician + reactive layer) — Hermes v0.11.x Discord bot
- Reads `~/context/status.md`, daily/weekly briefs, vault via obsidian MCP
- Discord channels: `#daily-brief` `#inbox` `#focus` `#executor` `#projects` `#journal` `#retrospector`
- Cron jobs: morning-brief (08:30 MTWRF), eod-summary (18:00 MTWRF), inbox-check (30 min 09–19 MTWRF), weekly-plan (Mon 08:00), weekly-review (Fri 17:00), retrospector (Fri 17:30), commit-vault (23:00 daily)
- Logs corrections to `corrections_log` table (SQL INSERT, silently on pushback)
- **Never accesses Asana API directly** — only reads pre-rendered markdown and DB
## Файлы конфигурации
- `~/.hermes/config.yaml` — Hermes config, модель, Discord, cron
- `~/.hermes/SOUL.md` — характер и правила Орла
- `~/.hermes/auth.json` — Claude Code OAuth (credential_pool)
- `~/Developer/personal-os/config.json` — Asana workspace/user GIDs
- `~/.config/personal-os/env` — env vars (ASANA_API_KEY, POSTGRES_URL)
### Ретроспектор (Retrospector) — Friday pattern analysis
- Runs as separate Hermes cron at 17:30 Friday → `#retrospector`
- Sources: corrections_log table, status.md subtask counts, obsidian git log, activity_daily quadrants, stories visibility metric
- Prompt: `~/Developer/personal-os/agent/prompts/retrospector.md`
### Исполнитель (Executor) — on-demand autonomous worker
- Triggered by explicit "fix bug [GID]" in #executor — never autonomous
- Creates git worktree at `~/DuckDuckGo/apple-browsers.git/.claude/worktrees/executor-{gid}-{slug}/`
- Single STOP gate after analysis; then autonomous: fix → build → test → draft PR → CI loop
- State machine in `executor_runs` table
- Prompt: `~/Developer/personal-os/agent/prompts/executor-bug-fix.md`
- For UI tests: spins VM via ddg-vm MCP (virfield), cleans up on PR close/merge
## Data Flow
```
Asana API
→ sync.js (30 min, Hermes cron, silent)
→ PostgreSQL: tasks, stories, task_edges, sync_state, task_annotations
ActivityWatch (local daemon)
→ generate-status.js (reads AW HTTP API)
→ activity_daily (upserts daily project/quadrant summaries)
generate-status.js
→ ~/context/status.md ← Eagle reads this
→ ~/Developer/personal-os/asana_context.md ← inbox-check reads this
Eagle corrections in Discord
→ corrections_log table (INSERT on pushback, no announcement)
→ Retrospector reads weekly
Executor runs
→ executor_runs table (state machine)
→ ~/DuckDuckGo/apple-browsers.git/.claude/worktrees/
→ GitHub draft PRs (--draft, --assignee @me, no reviewers)
```
## Key File Locations
| File / Path | Purpose |
|-------------|---------|
| `~/context/status.md` | Live Asana + AW snapshot. Written by generate-status.js. Eagle reads this. |
| `~/Developer/personal-os/asana_context.md` | Raw Asana context for inbox-check |
| `~/Developer/personal-os/agent/prompts/` | All agent prompts (Hermes cron + on-demand) |
| `~/.hermes/SOUL.md` | Eagle's identity, rules, vault write permissions |
| `~/Developer/personal-os/config.json` | Asana workspace/user GIDs, section GIDs |
| `~/Developer/personal-os/aw-projects.json` | ActivityWatch project classification rules |
| `~/.hermes/config.yaml` | Hermes config: model, MCP servers, cron settings |
| `~/obsidian/` | Vault — git repo, NAS remote, obsidian-mcp for search |
| `~/Developer/personal-os/logs/` | heartbeat.log, wiki-ingest-YYYY-MM-DD.md |
| `~/Developer/personal-os/briefs/` | daily/, weekly/, inbox/ — agent-written brief files |
| `~/DuckDuckGo/apple-browsers.git/` | Browser repo (bare), worktrees as sibling dirs |
| `~/DuckDuckGo/apple-browsers.git/.claude/worktrees/` | Executor worktrees |
## Database Tables
| Table | Written by | Read by |
|-------|-----------|---------|
| tasks | sync.js | generate-status.js, executor, status queries |
| stories | sync.js | generate-status.js, executor, retrospector |
| task_edges | sync.js | subtask traversal |
| sync_state | sync.js | generate-status.js header |
| task_annotations | manual / Eagle | generate-status.js irrelevant filter |
| activity_daily | generate-status.js | retrospector quadrant drift |
| corrections_log | Eagle on pushback | retrospector pattern analysis |
| executor_runs | executor | generate-status.js active runs, executor state machine |
## MCP Servers
**Hermes agents:**
| Name | Command | Used for |
|------|---------|---------|
| obsidian | `/opt/homebrew/bin/obsidian-mcp ~/obsidian` | Vault search, read, write |
**Claude Code / `claude -p` sessions:**
| Name | Command | Used for |
|------|---------|---------|
| ddg-vm | `npx tsx ~/Developer/virfield/server/mcp-server.ts` | VM lifecycle for UI tests |
| obsidian | `/opt/homebrew/bin/obsidian-mcp ~/obsidian` | Vault access |
## launchd Agents
| Label | Schedule | Purpose |
|-------|----------|---------|
| personal.os.heartbeat | :05 every hour | Watchdog — triggers missed Hermes jobs after wake |
| personal.os.wiki-ingest | 22:00 daily | `claude -p` wiki synthesis (needs filesystem access) |
## Vault Structure
```
~/obsidian/
├── wiki/ ← LLM-generated + hand-written (confidence: 1.0 = never overwrite)
│ ├── personal-os-architecture.md (this file)
│ ├── personal-os-purpose.md
│ ├── personal-os-agent-rules.md
│ ├── personal-os-self-modification.md
│ ├── personal-os-sync-pipeline.md
│ ├── personal-os-schema.md (auto-generated)
│ ├── ddg-asana-workflow.md
│ └── vault-filling-guide.md
├── raw/ ← symlinks to external source files (schema.sql, etc.)
├── work/
│ └── wiki/
│ └── apple-browsers/ ← .cursor/rules copies (.md) + vm-ui-testing.md
└── personal/ family/
```
wiki/personal-os-purpose.md
+73
View File
@@ -0,0 +1,73 @@
---
namespace: work
tags: [system, purpose, adhd, design]
last_updated: 2026-04-28
confidence: 1.0
---
# Personal OS — Purpose
## Why It Exists
Alex is a macOS Browser Developer / DRI at DuckDuckGo. Async-first company, primary channels Asana and Slack. ADHD profile.
The Personal OS exists to hold work context **outside Alex's head** — so that context isn't lost during deep focus, transitions, or when switching back from a meeting.
Without it, three things happen regularly:
1. A comment sits on an Asana task for days — Alex never saw it because he was in Xcode
2. A career-level task (Ship Review, Tech Design review) gets deferred until it becomes a bottleneck
3. End of day: "what did I actually do today?" — no clear answer, no signal for tomorrow
## Design Principles
**Principle 1 — Context is the product, not notifications.**
The system's job is to maintain a live, accurate picture of what's happening. Eagle reads that picture and speaks when something actually matters. It is not a notification firehose.
**Principle 2 — Protect focus.**
If Alex is in Xcode or Cursor, don't interrupt unless there's a red signal older than 3 hours. ADHD makes context switches expensive — the system should absorb noise, not amplify it.
**Principle 3 — No hallucinated state.**
Eagle never guesses about Asana tasks. All Asana data flows through sync.js → PostgreSQL → generate-status.js → status.md. Eagle reads markdown, not the API. This means the data is always consistent and never stale by more than 30 minutes.
**Principle 4 — Career goals are first-class.**
The system explicitly tracks: Project Advisor opportunities, visibility gaps (Ship Reviews where Alex hasn't commented), Tech Design reviews pending. These surface in status.md and the daily brief. Without explicit tracking they'd be invisible.
**Principle 5 — No new infra.**
The system has enough moving parts. New capabilities go into existing components: Postgres tables, generate-status.js queries, prompt files in `agent/prompts/`, SOUL.md rules. No new YAML files, no new shell scripts, no new launchd plists unless there is no other way.
## The Four Agents
| Agent | Role | When |
|-------|------|------|
| **Стратег (Strategist)** | Data pipeline — reads Asana + ActivityWatch, writes status.md | Every 30 min, workdays |
| **Орёл / Eagle (Tactician)** | Discord-facing reactive agent — briefs, inbox, focus gating | Cron + on-demand |
| **Ретроспектор (Retrospector)** | Weekly pattern analysis — corrections, quadrant drift, visibility | Friday 17:30 |
| **Исполнитель (Executor)** | On-demand bug-fix worker — analysis → fix → PR → CI loop | Explicit trigger only |
Eagle is the only agent Alex directly talks to. The others are background infrastructure.
## What "Work Context" Means
The system tracks four categories, mapped to `activity_daily.quadrant`:
| Quadrant | What it covers | Why it matters |
|----------|---------------|----------------|
| `project` | The current top-priority feature/bug | Time here should dominate most weeks |
| `aor` | AOR maintenance: code review, bug fixes, blocklist | Required to keep the area healthy |
| `career` | Project Advisor, Ship Reviews, Tech Design, visibility | Easily deferred; system forces it into view |
| `strategy` | Planning, Personal OS, Asana organisation | Meta-work; needs a floor, not a ceiling |
Personal pet projects (AXPressDeck, media_files_db, etc.) are tracked separately and not counted as work time. See `aw-projects.json` (`namespace: "personal"` entries).
## What It Does Not Do
- Does not make decisions for Alex — it informs
- Does not send notifications during focus (deep focus gate in inbox STEP 0)
- Does not access Asana API directly from Eagle — only reads pre-rendered status.md
- Does not create PRs, commits, or messages without explicit confirmation (except Executor, after its single approval gate)
## See Also
- `personal-os-architecture.md` — full technical architecture, agents, data flow, file locations
- `personal-os-agent-rules.md` — Eagle's operating rules and allowed writes
- `personal-os-self-modification.md` — how Eagle (and Alex) can evolve the system
wiki/personal-os-self-modification.md
+152
View File
@@ -0,0 +1,152 @@
---
namespace: work
tags: [system, self-modification, eagle, meta]
last_updated: 2026-04-28
confidence: 1.0
---
# Personal OS — Self-Modification Guide
How Eagle (and Alex via Claude Code) can safely evolve the system without breaking it.
**Rule**: Always show a diff/draft first, confirm, then apply. Commit the vault after.
---
## What Eagle Can Change (no approval needed — show draft first)
### 1. Prompt files (`~/Developer/personal-os/agent/prompts/*.md`)
Eagle reads and writes its own prompts. When Alex says "add X to the morning brief" or "change how inbox triage handles Y":
1. Read the current prompt: `~/Developer/personal-os/agent/prompts/<name>.md`
2. Draft the change, show the diff in Discord
3. On confirmation: write the file
4. Commit: `cd ~/Developer/personal-os && git add agent/prompts/<name>.md && git commit -m "[YYYY-MM-DD] prompt: <description>"`
**Do not** change `executor-bug-fix.md` without explicit confirmation — it controls autonomous code changes.
### 2. SOUL.md (`~/.hermes/SOUL.md`)
Eagle's identity, rules, and operating context. Eagle can propose changes to:
- Allowed vault write paths
- Focus gating rules
- Channel routing
Same flow: draft → confirm → write → no vault commit needed (SOUL.md is outside vault).
### 3. ActivityWatch project rules (`~/Developer/personal-os/aw-projects.json`)
To classify a new project as work or personal:
1. Read current `aw-projects.json`
2. Add entry with `project`, `namespace` (omit for work, `"personal"` for pet projects), and `rules`
3. Show draft, confirm, write
4. Commit: `cd ~/Developer/personal-os && git add aw-projects.json && git commit -m "[YYYY-MM-DD] aw: classify <project>"`
The `namespace: "personal"` field excludes a project from work activity in status.md and activity_daily.
### 4. Vault knowledge pages (`~/obsidian/wiki/`, `~/obsidian/work/wiki/`)
Eagle does **not** write to `wiki/` — that's the wiki-ingest job (22:00 launchd). But Eagle **can** write to:
- `~/obsidian/work/projects/SLUG.md` — project notes
- `~/obsidian/work/decisions/YYYY-MM-TOPIC.md` — decisions
- `~/obsidian/personal/` — personal notes
- `~/obsidian/family/` — family docs
After writing: `cd ~/obsidian && git add -A && git commit -m "[YYYY-MM-DD] <description>"`
### 5. Correction logging (corrections_log table)
Eagle silently logs a row whenever Alex pushes back on a suggestion. No approval needed — this is continuous background instrumentation, not a visible change.
```sql
INSERT INTO corrections_log (date, week_number, source, original_plan, correction, deferred_gids, reason_tag)
VALUES (current_date, EXTRACT(WEEK FROM current_date)::int,
'<source>', '<original>', '<user_message>',
ARRAY[<gids>]::text[], '<reason_tag>');
```
---
## What Requires Approval (always confirm explicitly)
### Schema changes (PostgreSQL)
New tables, columns, or indexes require a SQL migration. Pattern:
1. Draft the SQL in Discord, explain the purpose
2. On "go": run via `psql personal_os -c "..."`
3. Update `~/obsidian/wiki/personal-os-schema.md` (this is auto-generated at 22:00, but a manual update is fine)
Current tables: tasks, stories, task_edges, sync_state, task_annotations, activity_daily, corrections_log, executor_runs.
### New Hermes cron jobs
Add to `~/.hermes/config.yaml`. Format:
```yaml
crons:
- id: my-new-job
schedule: "30 9 * * 1-5"
prompt: |
Run: ~/Developer/personal-os/agent/prompts/my-prompt.md
channel: "#channel-name"
```
Must confirm before writing — a broken cron syntax silently prevents Hermes from starting.
### New launchd agents (`~/Library/LaunchAgents/`)
New plist files. Rarely needed — only if a job requires Mac filesystem access and can't run inside Hermes (e.g., wiki-ingest). Confirm before writing and before loading with `launchctl`.
### Executor prompt (`executor-bug-fix.md`)
Controls autonomous code changes. Changes here need explicit approval because a mistake could cause the Executor to behave incorrectly on real PRs.
### Hermes model or MCP configuration (`~/.hermes/config.yaml`)
Model upgrades, new MCP servers. Show the diff; confirm before writing.
---
## How generate-status.js Gets Extended
When a new data signal should appear in status.md:
1. Add a SQL query function (e.g., `getNewSignal()`)
2. Add a render function (e.g., `renderNewSignal(data)`)
3. Add both to `main()`: Promise.all for the query, the render call in `parts`
4. Test: `node ~/Developer/personal-os/generate-status.js` and read `~/context/status.md`
5. Commit
**Never** edit generate-status.js to change how Asana sync works — that's sync.js territory.
---
## Debugging the System
| Symptom | Check |
|---------|-------|
| status.md is stale (> 40 min old) | `stat -f "%Sm" ~/context/status.md` — if old, run `bash ~/scripts/run-pipeline.sh` |
| Hermes cron didn't fire | Check heartbeat: `tail -20 ~/Developer/personal-os/logs/heartbeat.log` |
| Eagle gave wrong Asana data | The DB may be stale — run `node ~/Developer/personal-os/sync.js` manually |
| activity_daily missing today | generate-status.js runs upsert — check `select * from activity_daily where bucket_day = current_date` |
| corrections_log empty | Eagle only inserts on pushback — expected to be sparse |
| Executor in stuck state | `SELECT id, state, updated_at FROM executor_runs WHERE state NOT IN ('complete','abandoned');` |
---
## Safe Operations Checklist
Before making any system change:
- [ ] Read the current file/config before proposing a change
- [ ] Show the diff, not just a description
- [ ] Wait for explicit confirmation
- [ ] Apply the change
- [ ] Verify: run the component or check output
- [ ] Commit if in a git repo (`personal-os` or `obsidian`)
---
## See Also
- `personal-os-architecture.md` — full system map
- `personal-os-agent-rules.md` — Eagle's rules and channel routing
- `personal-os-purpose.md` — why this system exists
work/wiki/apple-browsers/ci-pipeline.md
+222
View File
@@ -0,0 +1,222 @@
---
source: ~/DuckDuckGo/apple-browsers.git/main/.github/workflows/
confidence: 1.0
namespace: work
last_updated: 2026-04-28
tags: [ci, github-actions, xcodebuild, testing, build]
---
# CI Pipeline — macOS Browser
Reference for the GH Actions CI setup and how to run equivalent checks locally.
**Xcode version**: `26.4` (from `.xcode-version`)
**Working directory for all macOS jobs**: `macOS/` inside the repo
---
## Workflows Overview
| Workflow file | Trigger | What it does |
|---------------|---------|--------------|
| `macos_pr_checks.yml` | PR + push to main/release/hotfix | Unit tests, integration tests, release build, translations |
| `macos_ui_tests.yml` | PR (BSK changes) + push + schedule (3AM UTC) | Full UITest suite across macOS 14/15/26 |
| `macos_performance_tests.yml` | PR + push + daily 5AM UTC | Performance benchmarks (notarized build required) |
| `macos_pir_end_to_end_tests.yml` | PR + push + schedule | PIR feature E2E tests |
| `macos_build_notarized.yml` | Called by other workflows | Builds notarized Review .app + optional DMG |
| `macos_release.yml` | Manual + release process | Full release build and publish |
| `macos_check_sparkle_update.yml` | Post-release | Validates Sparkle update feed |
---
## PR Checks (`macos_pr_checks.yml`)
Runs on every PR. Three parallel tracks:
### Track 1: Unit + Integration Tests
Two matrix flavors run simultaneously:
| Flavor | Scheme | Runner | active-arch |
|--------|--------|--------|-------------|
| Non-Sandbox | `macOS Browser` | `macos-26` | YES |
| Sandbox | `macOS Browser App Store` | `macos-26-xlarge` | NO |
**Unit tests command:**
```bash
cd macOS
set -o pipefail && xcodebuild test \
-scheme "macOS Browser" \
-derivedDataPath "DerivedData" \
-configuration "CI" \
-skipPackagePluginValidation -skipMacroValidation \
-test-timeouts-enabled YES \
-default-test-execution-time-allowance 60 \
-maximum-test-execution-time-allowance 120 \
ENABLE_TESTABILITY=true \
ONLY_ACTIVE_ARCH=YES \
COMPILATION_CACHE_ENABLE_CACHING=YES \
"-skip-testing:Integration Tests" \
2>&1 | xcbeautify
```
**Integration tests command** (same flags, opposite -testing: flag):
```bash
"-only-testing:Integration Tests" \
-retry-tests-on-failure \
```
### Track 2: Release Build
```bash
cd macOS
set -o pipefail && xcodebuild \
-scheme "macOS Browser" \
-derivedDataPath "DerivedData" \
-configuration "Release" \
-skipPackagePluginValidation -skipMacroValidation \
COMPILATION_CACHE_ENABLE_CACHING=YES \
2>&1 | xcbeautify
```
### Track 3: ShellCheck + Bats (shell scripts)
```bash
# Bats shell tests
brew install bats-core
cd macOS && bats --formatter junit scripts/tests/*
```
---
## UI Tests (`macos_ui_tests.yml`)
### How it works in CI
1. **Build notarized app first** (`macos_build_notarized.yml`) — runs as a dependency
2. **Setup**: lists all `UITestCase`-derived classes from `UITests/` directory
3. **Matrix**: each test class × each macOS version runs as a separate job
4. **Retry**: `-test-iterations 2 -retry-tests-on-failure -test-repetition-relaunch-enabled YES`
### Running locally (two approaches)
**Option A — Direct xcodebuild (host only, no VM):**
```bash
cd ~/DuckDuckGo/apple-browsers.git/main/macOS
# Build for testing first
set -o pipefail && xcodebuild build-for-testing \
-scheme "macOS UI Tests CI" \
-derivedDataPath DerivedData \
-skipPackagePluginValidation -skipMacroValidation \
2>&1 | xcbeautify
# Run a specific test class
defaults write com.duckduckgo.macos.browser.review moveToApplicationsFolderAlertSuppress 1
set -o pipefail && xcodebuild test-without-building \
-scheme "macOS UI Tests CI" \
-derivedDataPath DerivedData \
-skipPackagePluginValidation -skipMacroValidation \
'-only-testing:UI Tests/AutocompleteTests' \
2>&1 | xcbeautify
```
**Option B — Via VM (virfield/ddg-vm MCP, matches CI environment):**
```
# 1. Build on host first (DerivedData must exist)
xcodebuild build-for-testing -scheme "macOS UI Tests CI" -derivedDataPath DerivedData ...
# 2. Start VM session
vm_prepare_session() → { vmName, ip }
# 3. Run tests in VM
run_tests(vm_id: vmName, scheme: "DuckDuckGo macOS", workspace: "<VMShare path>")
# 4. Poll
get_test_run_status(vm_id: vmName, run_id: run_id)
get_test_results(vm_id: vmName, run_id: run_id)
# 5. On failure: inspect
peekaboo_image(vm_id: vmName)
get_crash_reports(vm_id: vmName)
# 6. Always clean up
vm_stop(vm_id: vmName)
vm_delete(vm_id: vmName)
```
See `vm-ui-testing.md` for full ddg-vm MCP reference.
### Key notes
- All UI test classes **must** inherit from `UITestCase`, not `XCTestCase` — CI enforces this
- CI uses a notarized **Review** build (`DuckDuckGo Review.app`), not Debug
- Screen resolution is set to 1920×1080 in CI (macOS 15+) — VM does the same
- Build on host before running in VM — `run_tests` uses the pre-built xctestrun from `DerivedData`
---
## Executor Build Command (for Executor agent)
The Executor uses Debug configuration without code signing or notarization:
```bash
cd ~/DuckDuckGo/apple-browsers.git/.claude/worktrees/executor-{gid}-{slug}/macOS
set -e -o pipefail && xcodebuild \
-workspace DuckDuckGo-macOS.xcworkspace \
-scheme "DuckDuckGo macOS" \
-configuration Debug \
build \
2>&1 | xcbeautify
```
Key differences from CI:
- **No** `-derivedDataPath` override — uses default `DerivedData` in worktree
- **No** `ONLY_ACTIVE_ARCH` — defaults to YES for Debug, faster
- **No** `COMPILATION_CACHE_ENABLE_CACHING=YES` — not needed for single-run
- **No** `-skipPackagePluginValidation` unless you hit plugin errors (add if needed)
---
## Xcode Scheme Reference
| Scheme | Purpose | Used for |
|--------|---------|---------|
| `macOS Browser` | Non-Sandbox (direct distribution) | Unit tests, release build |
| `macOS Browser App Store` | Sandbox variant | Unit tests (sandbox) |
| `macOS UI Tests CI` | UI test runner | UI tests (requires notarized Review app) |
| `DuckDuckGo macOS` | Development scheme | Executor Debug build |
---
## CI Runner Reference
| Runner | Used for |
|--------|---------|
| `macos-26` | Unit tests (Non-Sandbox), release build |
| `macos-26-xlarge` | Unit tests (Sandbox), UI tests (latest macOS) |
| `macos-15-xlarge` | UI tests on macOS 15 |
| `macos-14-xlarge` | UI tests on macOS 14 |
| `ubuntu-latest` | ShellCheck, translation checks, Asana tasks |
---
## Common Local Equivalents
| CI step | Local command |
|---------|--------------|
| Select Xcode version | `sudo xcode-select -s /Applications/Xcode_26.4.app` (check `.xcode-version`) |
| Sync code signing | `cd macOS && bundle exec fastlane sync_signing_ci` |
| Build + unit test | See "Unit tests command" above |
| Build for UI test | `xcodebuild build-for-testing -scheme "macOS UI Tests CI" ...` |
| Run one UI test class | `xcodebuild test-without-building ... '-only-testing:UI Tests/ClassName'` |
| Check private API usage | `cd macOS && ./scripts/find_private_symbols.sh "DerivedData/Build/Products/..."` |
| Verify npm bundles | `npm run rebuild-autoconsent --workspace=macOS` |
---
## Cross-Reference
- `vm-ui-testing.md` — full virfield/ddg-vm MCP reference for VM-based UI tests
- `testing.mdc` → `testing.md` — UITestCase patterns, feature flag setup in tests
- `development-commands.mdc` → `development-commands.md` — team-approved xcodebuild flags