0 Fork 0

Files

T

Alexey Martemyanov f1828051d0 [2026-10-15] eagle: personal/projects/balda/execution-broker.md personal/projects/personal-os/personal-os-status-2026-10.md personal/projects/virfield.md personal/tech/agents-bridge.md work/projects/cpm-telemetry-data-requirements-2026-10-15.md work/projects/macos-cloud-autofix-plan-2026-10-15.md

2026-10-15 21:40:11 +06:00

32 KiB

Executable File

Raw Blame History

title, date, status, tags

title

date

status

macOS: cloud flow for automatically fixing and validating bugs and tests

Status: proposal, not implemented. Compiled on 2026-10-15 from: Asana (AI Agent Infrastructure / O-T), Obsidian (Virfield, Personal OS, macos-tests-worktrees-2026-10-14, ci-pipeline), .github/workflows and .claude/skills in apple-browsers, and Anthropic / OpenAI / Sentry / Bitrise documentation.

Legend: ✅ confirmed by a source, ⚠️ not verified / needs a spike.

1. What already exists (so we don't build it twice)

1.1 In Asana (company experience)

What	Summary	Lesson for us
Windows crash triage (1214714826035263)	A Prefect job pulls Sentry via MCP → Claude agent → daily stability report → Release DRI. ~~$8/day (~~$250–500/month). Started on Cursor self-hosted, then moved to Claude on a GitHub-hosted Windows runner that Prefect triggers as a GHA. A Prefect variable stores "memory" between runs, so cost drops day over day.	A working pattern: pre-fetch in a trusted zone → artifact → agent on the runner. The agent does not call Sentry itself.
Same task: developer attitude to auto-PRs	In April auto-PRs were met with hostility ("reviewing a PR costs more than checking the idea"). The fix: every finding is validated with a unit/UI reproducer test, and the report is root cause + direction + test. After that 2 AI PRs landed with no review comments, the team agreed to auto-PRs, and ~20 CPMN of degradation was prevented.	Do not start with auto-PRs. Trust ladder: report → proposal → draft PR. No fix is proposed without a failing test.
Find an alternative to Cursor Cloud Agents (1216757783325568, DRI Chris Clarke)	3 workflow classes: analyze-and-report / build-and-fix / drive-browser. Interactive Claude runners for Android, SERP and generic work already exist (claude.ai/code). Linux/Windows self-hosted GHA runners are in progress (1218221113915359). macOS is the main gap: Azure has no macOS, so there is no path to the internal network (dub, ClickHouse, Sentry). The "Provision Mac Machines" project (1215557028847084, AWS EC2 Mac) is ON HOLD.	We are closing exactly this gap. We need to plug into the AI Agent Infrastructure project rather than run a parallel one.
Context-transfer PoC (1217164276206833)	A GHA run is not resumable, so a ZIP artifact holds transcript + prompt + metadata, and a skill restores the context in a new chat. Proposed as a company-wide GHA.	Solves "a human wants to step in" without interactive sessions.
Bitrise (1218829288858490)	The Apple team already uses Bitrise as the macOS runner provider for GHA (GitHub macOS runners are too slow); GHA has a 360-minute cap. Bitrise RDE (remote dev environment: SSH/VNC, suspend/resume) is in trial: 1 concurrent session, and a Bitrise GitHub App bug with private repos (Bitrise is fixing it).	For automated runs use what already works (GHA + Bitrise). RDE is for interactive work.
Security triage: Bitrise RDE (1219413380003457, reply 2026-10-14)	(AT1) Direct access from Bitrise to internal systems is not viable (all Bitrise address pools would have to be opened). (AT2) Use the RDE MCP with a local Claude as orchestrator; set `SSH_AUTH_SOCK` to empty (it forwards keys). (AT3) Do not auto-approve RDE tools in a session that also has the ClickHouse/Sentry MCP; anything coming back from RDE is untrusted. RDE sessions themselves are isolated.	Hard design constraint: an agent on a Bitrise machine cannot see Sentry/ClickHouse/dub. Data arrives as an artifact.
Bug Squisher / Duck.ai	SERP: `[🤖 Bug Squisher]` PRs in ddg. Duck.ai: an engineer manually runs `$auto fix this bug` (a pstack port) → a ready PR → review. Automation comes later, once reliability is proven. Artur (Chromium): the agent does not drive the browser directly but writes a reproducible journey script that is run on the base and fix commits, so the agent cannot "lie" in validation.	Reuse this trick for validation (see §5). Align with flux/pstack so we don't add a fifth parallel framework (see the overlap comment in 1213572946361338).
Gate model (same discussion)	Consensus gates (kick-off, tech design, post-mortem) belong to humans; an unattended agent skips them. Review gates (privacy/security triage, ship review) are mandatory — they stack at the end, and corrections send the agent back into the loop.	Agent flow = draft PR + mandatory review gates, no consensus gates.
Windows release quality (1215834564874913)	Windows is the pilot; macOS follows. Preview catches crashes, but they are not turned into bugs with an owner (a "conversion problem").	Our flow is the "converter" crash → reproduced test → draft PR for macOS.

⚠️ I found no separate Asana task about Android cloud AI bug fixing — only the mention that Claude runners are available for Android/SERP/generic. If there is a specific link, it needs reading. ⚠️ There is no "Sentry crash report → Claude cloud" flow for Apple either: today it is the manual ddg-sentry-report skill (the user invokes it, it writes to Asana).

1.2 In the apple-browsers repo

macos_pr_checks.yml (unit + integration on macos-26 / macos-26-xlarge, release build, retry), macos_ui_tests.yml (class × macOS 14/15/26 matrix, nightly 03:00 UTC, workflow_dispatch with single_test_class, user_modes, workflow_call), macos_run_ui_test.yml, macos_build_notarized.yml (UI tests need the notarized Review app), danger.yml, conflict_watch.yml, .github/actions/bitrise-build-cache, process-test-results, asana-failed-pr-checks.
Skills: ddg-diagnose-ci-failure (JUnit XML from artifacts, classification, flaky vs deterministic, regression timeline) and ddg-sentry-report. Both are user-initiated: they touch Asana, so the "lethal trifecta" policy blocks other MCPs afterwards.
AGENTS.md, CLAUDE.md, REVIEW.md already exist.
.xcode-version = 26.4 (ci-pipeline note). The Virfield profile uitest-27-v1 = macOS 27 / Xcode 27, so there is a mismatch; we need an image for 26.4.

1.3 Our local achievements (Obsidian)

Virfield (Tart + SSH nodes, Go, MCP with 18 tools: vm_acquire/vm_lease/vm_release/vm_tunnel, image_*):

✅ Disposable macOS VM per lease, pinned SSH, artifact export, cleanup; goldens for Xcode 15.2 / 13.4.1 / 27.0.
✅ Fixed the things DDG UI tests fail on: locked keychain (login + default, GUI and SSH contexts), 1920×1080, ASIF/Recovery, SSH banner timeout. A real DownloadsUITests.testDownloadFinishesThenPopupIsShown ran twice with 0 failures (Xcode 27.1 products → guest 27.0).
⚠️ Not accepted yet: Broker/Runner → the full apple-browsers UI suite; installed aee89c1 ≠ source 9a740c0; false vm_drift events; admission quirks (storage reserve); pool = 2 VMs on one Mac; GitHub CI deferred. It is a good dev/acceptance sandbox, not a production fleet.

Personal OS / Executor (3 attempts, 2026-04…10):

Executor v1 (autonomous LLM orchestrator) → incident (wrote to Asana, task data leaked to Discord). Executor v2 (bash runner/analyzer + 5 worker prompts: queued / in_progress / fix_ci / fix_comments / pr_review; state machine queued→in_progress→pr→complete/abandoned; no-output watchdog 30 min, hard limit 3 h; lockfile; analyzer filters) was implemented, but the crons are paused.
Cross-cutting lessons: (1) the deterministic layer worked, the agentic one did not; (2) an agent must not write to external systems or have an outbound channel carrying work data, and human-in-the-loop is a mandatory gate; (4) weak hardware cannot carry an agent stack; (7) the model provider is config, not architecture; Codex bwrap failed under the default Docker isolation; the personal Broker later accepted Story/restart/denied-write on its opt-in nested-v1 host/profile with enforced AppArmor and seccomp (2026-10-15; personal/projects/balda/execution-broker). That scoped result does not qualify another host or this proposed company flow. Do not proxy subscription OAuth tokens.

Test fixes of 2026-10-14 (work/projects/macos-tests-worktrees-2026-10-14) — a ready quality reference and eval set:

AddressBarTests: root cause = ensureObjectDeallocated(.interrupt) sends SIGINT while an object is still alive; two causes (autorelease in TestRunHelper, strong self in an animation completion). Validation: main 10/10 FAIL → fix 15/15 PASS, 7/7 isolated, full Integration ×2.
Fire Window video leak: not reproduced; outcome = 4 guard tests, no fix (validity check: an intentional leak makes the test fail). "Not reproduced" is a legitimate outcome.
Environment observations: display sleep breaks dealloc checks (display-link animations stall), so the VM/runner must keep the display awake; async tests mask releases; network mocking (OHHTTPStubs) and isolation of user settings are required.

1.4 External (checked against documentation)

Platform	What it does	Limitation for us
`anthropics/claude-code-action@v1`	Automation mode (`prompt`); triggers: schedule / dispatch / issue / PR; `claude_args` (`--max-turns`, `--allowedTools`, `--model`); skills from `.claude/skills`; auth: API key, OAuth, OIDC federation (no long-lived secrets); by default creates a branch and the PR is created via a link (a human clicks). Commits made with `GITHUB_TOKEN` do not trigger CI, so a GitHub App/custom token is needed.	⚠️ The docs say "runs on GitHub-hosted runners"; running on macOS/a Bitrise label is not explicitly described → spike. Nothing prevents using `claude -p` directly in a job.
`openai/codex-action@v1`	`codex exec` in a job; Linux or macOS runner; `sandbox`, `safety-strategy: drop-sudo`, `allow-users`.	No fix-CI examples (only PR review).
Claude Code cloud (claude.ai/code, routines)	Sessions on Anthropic VMs; routines: schedule / API trigger (`POST /fire`, the `text` field = alert payload) / GitHub events; Auto-fix PR (reacts to CI failures and comments).	Anthropic VMs do not build Xcode. Routine limits: 30 fires/hour per routine. `text` is wrapped as untrusted.
Claude self-hosted environments (public beta, Team/Enterprise)	A runner on our host; outbound-only; routines and claude.ai/code route to it; ephemeral runner (`--drain-grace-sec 0` → clean disk); sessions are resumable from any device. A runner locks to one owner.	⚠️ A macOS runner is not described in the docs (recipes are Kubernetes/Compose); Chris Clarke: "supports macOS, but we don't have any provisioned". Ask Anthropic (the project has a similar subtask for Windows).
Sentry Seer	Automatic Issue Fix: ≥10 events, activity within 14 days, fixability score; levels: stop after RCA / solution / PR / hand-off to Claude Agent (or Cursor, Copilot); Seer never merges. They recommend starting with "Stop after Solution".	Does not build the macOS project; needs the GitHub integration + code mappings; privacy (crash data → an external agent) needs a privacy triage.
Bitrise Build Hub (GHA)	`runs-on: <bitrise-label>`, M4 Pro, Xcode within 24 h of release, on-demand ephemeral with the GitHub App, US/EU.	No explicit egress-control guarantees.
Bitrise RDE	SSH/VNC, suspend/resume, MCP server.	See the security triage above; trial = 1 session.
Bitrise AI Build Fixer	Analyzes a failed build → PR → rebuild (Pro/Enterprise).	⚠️ Described for Bitrise CI; not confirmed for "Build Hub under GHA". Covers only build failures, not flaky tests or crashes.
Tart / Orchard / Tartelet / Cilicon	OSS ephemeral macOS VMs on Apple silicon for self-hosted GHA runners.	⚠️ The news about Cirrus CI shutting down comes from a competitor's blog and is unverified; Tart is OSS and does not depend on it.

2. Problem statement

Goal: given a signal (a test fails/flakes on main; a new crash in Sentry; a bug), produce a draft PR with evidence that the fix works (before: red N/N, after: green N/N, lint, required schemes build), or a reasoned refusal ("not reproduced", "needs a human"). A human reviews and merges.

Non-goals: auto-merge; the agent writing to Asana/chats; fixing features/UX without an owner; release branches in the first stage.

Task classes (increasing complexity and risk):

Flaky/failing unit & integration test (plain macOS, no secrets).
Failing UI test (needs the notarized Review build, UITestCase, OS matrix).
Crash from Sentry (needs a reproducer; data comes from the internal zone).
Bug from Asana (needs repro steps; highest prompt-injection risk and risk of architectural mistakes).

3. Target architecture (shared by all options)

flowchart LR
  subgraph Signals["Signals"]
    A["CI: JUnit from main / nightly runs"]
    B["Sentry: new in release, N+ events"]
    C["Asana bug / manual trigger"]
  end
  subgraph Trusted["Trusted zone - Linux, has dub and Sentry access"]
    D["Deterministic intake:<br/>fingerprint, dedupe, eligibility,<br/>budget, PII scrub"]
    E["signal.json + artifacts"]
  end
  subgraph Mac["macOS runner - ephemeral, no internal network"]
    F["Agent job: analyze, repro, fix, validate"]
    G["Evidence bundle:<br/>xcresult, before/after, summary.json"]
  end
  H["Draft PR ai/fix/fingerprint<br/>label ai-generated"]
  I["Existing PR CI:<br/>unit + integration + UI on Bitrise"]
  J["Human: review and merge"]
  K["Notifier: Mattermost / Asana comment<br/>deterministic, narrow token"]
  A --> D
  B --> D
  C --> D
  D --> E --> F --> G --> H --> I --> J
  H --> K
  I -->|"CI failure"| F

Principles (from the Executor lessons and the security triage):

A deterministic skeleton, with the LLM only in a narrow worker step. Queue, dedupe, budget, retries and notifications are code, not an agent.
The agent gets no Asana, Sentry or ClickHouse access. Input is files (signal.json); output is a branch ai/fix/<fingerprint> + summary.json.
Write access only as a draft PR on an ai/* branch (GitHub App; ruleset: only ai/* may be pushed, main is closed). Notifications and Asana comments are done by a separate deterministic step.
Signing/notarization secrets are not in the agent job. The agent job runs unit/integration without secrets; UI tests are launched by the orchestrator on the agent's branch through the existing macos_run_ui_test (like any human PR).
Provider = config: agent: claude|codex, one prompt/skills/schema (AGENTS.md ↔ CLAUDE.md).
Input is untrusted (stack traces, bug texts, PR comments): pass it as data, minimal --allowedTools, no write MCPs.
Everything is idempotent: one fingerprint = one branch/PR; a retry does not create duplicates.

4. Execution options (venue)

	Description	Pros	Cons / risks	When
A. GHA + Bitrise Build Hub + `claude -p` / action (recommended start)	A workflow in apple-browsers: `dispatch/schedule` → Linux job (intake) → macOS job on a Bitrise label with the agent.	Already used for CI; nothing new to buy; ephemeral; artifacts and OIDC out of the box; same PR loop as humans; same Xcode as CI (key for reproducibility).	Atomic (no chat) → context-transfer ZIP; 360-min cap; no internal network (fine, data arrives as an artifact); runner egress is not controlled → in-VM sandbox (Seatbelt) and prompt hardening; ⚠️ claude-action on macOS is not documented → spike.	Phases 1–3. Classes 1, 2, 3.
B. Self-hosted Tart runners (Virfield as the image base)	Our own Mac minis + Tart + (Orchard/Tartelet/Cilicon) as GHA runners, an image with Xcode 26.4 from our image pipeline.	Full network control (egress allowlist → internal data possible); interactive access over an SSH tunnel; no per-minute cost.	Needs DDG hardware (the "Provision Mac" project is on hold; AWS EC2 Mac is expensive, with a 24 h dedicated-host minimum ⚠️); ops load; Virfield is not accepted and is currently a personal Mac with 2 VMs (bus factor 1).	Phase 4+, if A hits cost/network limits. Virfield stays a dev/acceptance sandbox.
C. Claude self-hosted environment on Mac (Tart VM per session)	An Anthropic runner on a macOS host; triggers: routines `POST /fire` (from Sentry/GHA), claude.ai/code.	Resumable interactive sessions from any device (closes A's main gap); routines/API trigger; ephemeral runner = clean disk.	Beta, Team/Enterprise, ⚠️ macOS unconfirmed; owner lock → needs a service account; privacy/security triage for Claude self-hosted is already marked "revisit"; depends on the Anthropic control plane.	Phase 4, after Anthropic answers. Interactive "human joins".
D. Bitrise RDE + Claude	A long-lived macOS VM with SSH/VNC and MCP.	Interactive, suspend/resume, screenshots/VNC.	Trial = 1 session, private-repo bug, no internal network, SSH agent forwarding (blank it), does not scale, orchestrator = local Claude.	One-off hard bugs / UI repro, not a pipeline.
E. Cloud Linux agents + CI as the oracle	Claude auto-fix PR / Seer hand-off / Codex cloud edit code without building; Bitrise CI validates.	Zero infrastructure; quick to start; good for lint/compile/obvious tests.	Cannot reproduce or debug flaky tests (no Xcode); every iteration is a full CI cycle (slow/expensive); auto-fix reacts to CI but does not replace a reproducer.	Complement to A: (1) Seer RCA as input, (2) auto-fix on already-open AI PRs.
F. Bitrise AI Build Fixer	A ready-made Bitrise feature.	Nothing to write.	⚠️ Build failures only and probably Bitrise CI only; Pro/Enterprise; closed logic/validation.	Not a base. Check it if build breakages are frequent.

Recommendation: A as the base + E as a complement (Seer RCA / auto-fix), then choose C or B for interactivity and cost reduction based on data. Virfield is for developing/accepting images and prompts locally, so we don't burn Bitrise minutes on debugging.

Claude vs Codex

One harness, two backends: scripts/ai-fix/run-agent.sh --agent claude|codex. Same input/output (signal.json → summary.json + branch).
Claude first (corporate keys/plugins, ddg-required-config, OIDC federation, our skills). Codex as an A/B comparison on the same eval set and insurance against a provider change. The Codex action supports a macOS runner and drop-sudo; Codex cloud (a Linux container) is not suitable for Xcode ⚠️ (the docs are silent about macOS).
Compare by the §8 metrics on identical tasks, not by feel.

5. Validation (the core of the plan)

5.1 Ladder by signal type

Type	Repro	A fix counts as valid if
Flaky unit/integration	`-test-iterations N` / `-run-tests-until-failure` on the same Xcode/OS, display awake (otherwise false dealloc flakes, see 10/14). Start from CI history (failure rate).	Red before (≥K of N), green M/M after (M ≥ 15–20); not masked (`async`); whole class + neighbours green; network/settings mocked; `swiftlint --strict` 0; schemes `macOS Browser` and `macOS Browser App Store` build.
Failing test on main	bisect against the last-green CI sha	Is the cause the test or the product? If the product — fix + test; afterwards the full unit suite.
Failing UI test	run `single_test_class` on the notarized Review build through the existing workflow	Green on the target OS; then the 14/15/26 matrix on request; `-retry-tests-on-failure` does not count as "green".
Crash (Sentry)	a mandatory reproducer (unit/UI/stress test). No reproducer → RCA + hypothesis only, no PR.	The test is red before the fix and green after; the symptom matches the stack trace; no new crashes in neighbouring tests.
Bug (Asana)	a journey script written by the agent	The same script run on base and on fix shows a difference; acceptance criteria from the ticket; a human confirms the direction before the PR (two-step mode).

5.2 Mandatory evidence bundle

summary.json by schema: signal_id, classification, root_cause, confidence, repro: {command, before: k/n fail, after: m/n pass}, validation: {schemes, lint, test_classes}, files_changed, outcome: fix | guard_tests_only | not_reproduced | needs_human, risks, commands_run, plus .xcresult/logs. The PR description is generated deterministically from this file.

5.3 Protection against "the agent lies"

Validation is done by a script, not by the LLM: the agent supplies a command/test, the harness runs it N times and counts itself.
Artur's trick: a single journey script for base and fix.
Mutation check (as on 10/14): deliberately reintroduce the bug → the test must fail.
A second agent in a clean context as reviewer per REVIEW.md (/code-review); the result is a PR comment, not a blocker.
Diff limits: file/line caps; changes to test infrastructure/assertions without reason → needs_human; weakening assertions and deleting tests are forbidden (as in the macos-uitest skill).

5.4 Eval set before enabling automation

Golden cases (known "correct" fix): the AddressBarTests flake (root cause in 2 places), the Ask AI Chat button animation leak, the DeallocationTests display-sleep crash, the Fire Window leak (expected outcome = not_reproduced), plus 5–10 historical fixes from git log / the Asana "Apple CI Failing Tests" project. Metrics: was the root cause found, is the fix valid, was a fix not invented where none exists. The threshold for enabling the schedule is agreed in advance.

6. Security and privacy

Threats and mitigations:

Prompt injection via Sentry/Asana/PR comments → inputs as data; minimal --allowedTools; no write MCPs; --max-turns budget; network denied except the allowlist.
Exfiltration: the agent has no application secrets; the GitHub App token is narrow (ai/*); OIDC instead of static keys; internal data only in signal.json after scrubbing (user data, URLs, PII) in a trusted Linux job; for the Windows version a privacy follow-up on blob transfer is already open, so open a similar triage for Apple.
Egress on Bitrise runners is not controlled → in-VM Seatbelt sandbox (Claude Code / Codex on macOS) + proxy allowlist (github.com, api.anthropic.com / api.openai.com, required registries/CDNs); in option B — pf on the host.
SSH forwarding (from the triage): do not forward SSH_AUTH_SOCK; in the RDE option — per AT2/AT3.
Signing: build-with-secrets and agent are separate jobs; UI tests on an ai/* branch go the same way as a human PR (ruleset + environment approvals for secrets).
Kill switch: vars.AI_FIX_ENABLED, a daily $ budget, maxActiveRuns, an allowlist of classes/paths (by default do not touch: signing, entitlements, Package.swift of external dependencies, release workflows, .github/**).
Audit: transcript + commands kept as an artifact for 30 days (like the Virfield journal).
Review gates: privacy triage for changes (if a PR touches pixels/privacy — Danger/privacy-review.yml already exist); the PR template requires an Asana link → the Asana task is created/linked by a human or a deterministic service, not by the agent.
Do not repeat: an autonomous LLM orchestrator; writes to Asana/Discord from the agent; proxying subscription OAuth tokens.

Approvals needed (in parallel with Phase 0): privacy + security triage for (a) a Bitrise runner + Anthropic/OpenAI API with apple-browsers code, (b) passing scrubbed Sentry data, (c) Claude self-hosted (if C).

7. Phased plan

Phase 0 — alignment (≈1 week, 2–3 person-days)

Join "AI Agent Infrastructure" (project) and talk to Chris Clarke: record that apple-browsers = build-and-fix on macOS via GHA + Bitrise (what the comments already agree on), not a new stack.
Sync with Artur/Sebastien/Tim (flux/pstack, verification) and Bárdi (Prefect crash flow, Windows runner) — take their harness/skills instead of writing our own; our contribution is the macOS specifics (Xcode, XCTest, display sleep, UITestCase).
Open the privacy/security triage (§6). Order a dedicated API key (the form from the tickets) or OIDC federation.
Baseline metrics: CI retry share, top-N flaky tests over 30 days (from JUnit), open "Apple CI Failing Tests".
Result: a decision on venue A, a stakeholder list, agreed guardrails.

Phase 1 — deterministic skeleton + report-only (≈2 weeks, ~8 person-days)

Workflow ai_triage_macos.yml (schedule + dispatch): collects JUnit from recent main/nightly runs (process-test-results, gh api — like ddg-diagnose-ci-failure), computes failure rate, fingerprints, dedupe, eligibility; publishes a report (Mattermost/job summary). No LLM writes.
signal.json / summary.json schemas; run-agent.sh (claude|codex), path allowlist, budget.
The §5.4 eval set + harness locally on Virfield (cheap prompt debugging on an Xcode 26.4 image; build the image with image_create if needed).
Spike S1 (claude/codex CLI on a Bitrise runner: install, auth, Seatbelt, cost per minute on a single test).
Exit: a daily report; the eval harness runs; S1 gave time/cost numbers.

Phase 2 — test agent: draft PR on request (≈3 weeks, ~12 person-days)

ai_fix_test.yml (workflow_dispatch: test_id, agent, mode): Linux intake → macOS agent job (analyze → repro N times → fix → validate) → evidence bundle → draft PR ai/fix/<fp> with a label.
A human triggers it manually (like $auto in Duck.ai). Then the existing PR CI; UI tests via a separate orchestrator dispatch.
A/B Claude vs Codex on the eval set.
PR-feedback loop: an @ai-fix comment or a repeat dispatch → same branch + context-transfer ZIP; optionally Claude auto-fix for CI failures on ai/* (option E).
Exit: ≥ the threshold on the eval (root cause found, no false fixes); 5 real PRs passed review; average cost per fix measured.

Phase 3 — crashes: Sentry → proposal → draft PR (≈3 weeks, ~12 person-days)

Pre-fetch in a trusted Linux job (Sentry MCP/API; projects apple-ios, apple-macos; filters firstRelease, app_version, events ≥ N; reuse the ddg-sentry-report logic) → scrub → signal.json.
In parallel a Seer trial on apple-macos in "Stop after Solution" mode (if privacy approves): use the RCA as an extra input/benchmark; compare with our agent.
Two-step mode by default: proposal (root cause + reproducer test + direction) → comment to the owner → after a "yes" the agent makes the PR. After trust accumulates (as in the Windows case) — automatic draft PRs for crash classes with a reproduced test.
Exit: ≥ X crash findings with a reproducer; share of directions accepted by owners.

Phase 4 — automation and interactivity (≈4–6 weeks, depending on results)

Enable the schedule (nightly top-K eligible), limits/budget, a metrics dashboard.
Choose interactivity by data: C (if Anthropic confirms macOS), B (if Bitrise minutes cost/network become a problem), or keep D for rare cases.
Asana bug fixes (class 4) only after proven reliability on 1–3 and together with flux/pstack.
A decision on release branches is separate.

Rough estimate through Phase 3: ~8–10 weeks elapsed, ~35–45 person-days (rough; depends heavily on triage and the S1 results).

8. Metrics and stop criteria

Acceptance rate: share of AI PRs merged without major rework.
Precision: share of fixes that survived production without regression; share of not_reproduced outcomes that were actually real (false negatives).
Cost per merged fix (tokens + Bitrise minutes), time-to-PR, reviewer-minutes per PR.
Flaky rate / CI retry rate on main before/after; for crashes — the CPMN/crash-free delta for affected signatures.
Stop conditions: acceptance below threshold two cycles in a row; any leak incident; budget exceeded → automatic kill switch.

9. Risks

Risk	Mitigation
The agent fixes the symptom (weakens an assertion, adds a sleep/retry)	Prompt prohibition + diff policy + the harness counts N/N itself + mutation check
False flaky fixes (the test was broken, not the product)	"test bug vs product bug" classifier + review agent + owner
Display sleep/environment behaves differently from CI	Same image/Xcode, `caffeinate`/display awake, compare with CI history
Xcode mismatch (26.4 vs 27)	Image strictly per `.xcode-version`; a 26.4 Virfield profile
Bitrise minutes/token cost	Budgets, top-K, nightly mode, a Sonnet-class model by default, caching (Bitrise build cache action)
Prompt injection via Sentry/Asana	Scrub + data-only + minimal tools + sandbox
Duplicating others' efforts (flux/pstack/Prefect/Bug Squisher)	Phase 0, a shared harness, our contribution is the macOS layer
Uncontrolled PR growth	One fingerprint = one PR, a cap on open AI PRs, stale auto-close
Dependence on personal infrastructure (Virfield on one Mac)	Virfield is not on the production path
Anthropic beta features (self-hosted, routines `experimental` header)	Not on the critical path until GA

10. Open questions (need decisions/data)

Confirm venue A: who is the DRI for macOS in AI Agent Infrastructure, and can a Bitrise runner label be allocated for agent jobs.
⚠️ S1: do claude -p / claude-code-action work on a Bitrise macOS runner (auth, Seatbelt, network)? What does a minute/task cost?
⚠️ Anthropic's answer: a macOS runner for self-hosted environments — timing and support.
Privacy/security: may scrubbed Sentry stack traces and apple-browsers code be sent to an agent on a Bitrise runner (OpenAI needs a separate approval).
Where the pipeline lives: apple-browsers (.github/workflows) vs a separate repo + a shared flux plugin.
Who owns the Asana linking of the PR (the template requires an Asana link): a deterministic service or a human.
Android: clarify whether a separate project exists (none found) — maybe the pattern can simply be reused.
Whether to run a pilot with "iOS/macOS unit flaky only" before crashes (I recommend yes).

11. First concrete steps (this week)

Write a short comment/task in AI Agent Infrastructure with this plan and questions 1–3 (without duplicating projects).
Assemble the eval set (4–5 cases from 10/14 + 5 from history) and describe the "expected outcome" for each.
Locally on Virfield (or a dev Mac) run claude -p on one case with the "N× before/after" harness — get a baseline of quality and cost before any CI integration.
Do spike S1 on a Bitrise runner in a separate branch (workflow_dispatch, a single test).
Prepare a draft privacy/security triage for (a) and (b).

32 KiB Executable File Raw Blame History Unescape Escape