# Plan: GHA CI Executor — AI-Powered Bug Fix Pipeline **Date**: 2026-05-14 **Status**: Planning **Owner**: Alex M ## Related - **Asana task: MM Bots**: https://app.asana.com/1/137249556945/project/908478224964033/task/1214799615211686 - **Mattermost Migration Plan**: [[mattermost-migration]] - **Executor Orchestrator Redesign**: [[executor-orchestrator-redesign]] - **Executor Orchestrator Redesign**: [[executor-orchestrator-redesign]] --- ## Goal Build an AI-powered bug fix pipeline that scales from personal use to org-wide deployment: - Tasks arrive from multiple sources (Asana, Sentry, `@claude` in PRs) - Analysis Agent generates a rich, context-aware prompt - GHA executes Claude Code on the right runner (macOS, Linux, self-hosted) - Results posted back to source (Asana task, Sentry issue, PR comment, MM channel) --- ## Architecture Three distinct layers, each independently replaceable: ``` ┌─────────────────────────────────────────────────────────┐ │ TRIGGER LAYER │ │ │ │ Asana task tagged "обработать" │ │ Sentry crash report [→ AI Analysis Placeholder] │ │ GitHub: @claude mention in PR/Issue (native, no glue) │ │ Eagle: on-demand ("go [GID]") │ └───────────────┬─────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ ORCHESTRATION LAYER — Prefect │ │ │ │ Flow: execute_task(source, task_id) │ │ 1. Analysis Agent (Claude) → structured prompt │ │ 2. Select execution target (repo + runner type) │ │ 3. Dispatch GHA workflow via `gh workflow run` │ │ 4. Monitor job status (poll / webhook) │ │ 5. On complete: post result to source + notify MM │ │ │ │ State: PostgreSQL | UI: Prefect dashboard │ │ Concurrency: max N in-flight | Retry: built-in │ └───────────────┬─────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ EXECUTION LAYER — GitHub Actions │ │ │ │ uses: anthropics/claude-code-action@v1 (official GA) │ │ │ │ Runner selection by task type: │ │ ├── macOS bug (iOS/macOS): macos-15 or self-hosted Mac │ │ ├── macOS visual repro: ddg-vm (Peekaboo inside VM) │ │ └── General (Linux): ubuntu-latest │ │ │ │ Output: draft PR + result artifact (JSON) │ └───────────────┬─────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ NOTIFICATION LAYER — Mattermost / Zulip │ │ │ │ executor-bot → #executor-queue (MM) │ │ Summary: task, PR link, test result, confidence score │ └─────────────────────────────────────────────────────────┘ ``` --- ## Analysis Agent Runs inside Prefect flow (on Eagle or serverless), before GHA dispatch: **Input:** task source (Asana GID / Sentry issue ID / GitHub issue URL) **Steps:** 1. Read full task context: description, comments, user reports, related code 2. Search codebase for relevant files (git grep, symbol search) 3. Generate structured executor prompt: - Problem statement - Reproduction steps - Investigation plan (files to check, hypotheses) - Acceptance criteria (what "fixed" looks like) - Test strategy (unit / UI / Peekaboo visual) 4. Classify task → select runner type **Output:** `executor_prompt.md` + runner classification + branch name --- ## Execution Layer: `claude-code-action@v1` Anthropic's official GA action (released 2025). Key advantages over manual `claude -p`: - Handles GitHub context natively (PR diffs, issue body, comments) - Auto-posts results as PR review comments - `--dangerously-skip-permissions` not needed — action has scoped GitHub token - Works with `workflow_dispatch` for external triggers ```yaml name: Executor Worker on: workflow_dispatch: inputs: task_gid: description: 'Asana task GID' required: true executor_prompt: description: 'Base64-encoded executor prompt' required: true branch_name: description: 'Git branch to create' required: true runner_type: description: 'macos-15 | ubuntu-latest | self-hosted' default: 'macos-15' jobs: executor: runs-on: ${{ inputs.runner_type }} timeout-minutes: 120 steps: - uses: actions/checkout@v4 with: ref: main fetch-depth: 0 - name: Decode prompt run: echo "${{ inputs.executor_prompt }}" | base64 -d > /tmp/executor-prompt.md - name: Run Claude Code uses: anthropics/claude-code-action@v1 with: prompt: /tmp/executor-prompt.md anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }} github_token: ${{ secrets.GITHUB_TOKEN }} claude_args: | --model claude-opus-4-7 --max-turns 30 - name: Post result to Asana if: always() env: ASANA_TOKEN: ${{ secrets.ASANA_CI_TOKEN }} run: | # Post PR link + test summary to Asana task ${{ inputs.task_gid }} # See scripts/post-executor-result.sh ``` --- ## Orchestration Layer: Prefect ### Why Prefect (not Temporal, not Kestra) | | Prefect | Temporal | Kestra | |---|---|---|---| | Learning curve | Low (Python decorators) | Very high (~1 month) | Medium (YAML) | | AI/agent workflows | ✅ Native Python | ⚠️ Lots of boilerplate | ❌ YAML gets messy | | Self-hosted | ✅ Docker, simple | ✅ Complex cluster | ✅ Docker | | Retry / state | ✅ Built-in | ✅ Indestructible | ✅ Kafka-backed | | Org-scale RBAC | Prefect Cloud (paid) | ✅ | ✅ | | **Fit for this use case** | **✅ Best** | ⚠️ Overkill | ⚠️ Wrong paradigm | Temporal is overkill: our tasks complete in <2h, don't need month-long replay guarantees. Kestra is designed for ETL/data, not AI agent orchestration. ### Prefect Flow Design ```python from prefect import flow, task import subprocess, base64, time @task(retries=2, retry_delay_seconds=60) def run_analysis_agent(task_source: str, task_id: str) -> dict: """Claude analyzes the task and returns structured executor prompt.""" # Runs Claude (hermes or claude -p) with task context # Returns: {prompt_b64, branch_name, runner_type, confidence} @task def dispatch_gha(repo: str, prompt_b64: str, branch: str, runner: str, task_gid: str) -> str: """Triggers GHA workflow_dispatch. Returns run_id.""" result = subprocess.run([ "gh", "workflow", "run", "executor-worker.yml", "--repo", repo, "-f", f"executor_prompt={prompt_b64}", "-f", f"branch_name={branch}", "-f", f"runner_type={runner}", "-f", f"task_gid={task_gid}", ], capture_output=True, text=True) return extract_run_id(result.stdout) @task(retries=60, retry_delay_seconds=60) def wait_for_gha(repo: str, run_id: str) -> dict: """Polls GHA job until complete. Returns result artifact.""" status = get_gha_status(repo, run_id) if status in ("in_progress", "queued"): raise Exception("Still running") # triggers retry return get_result_artifact(repo, run_id) @task def post_result(source: str, task_id: str, result: dict): """Posts PR link + summary back to Asana/Sentry/MM.""" ... @flow(name="executor-task") def execute_task(source: str, task_id: str, repo: str = "duckduckgo/apple-browsers"): analysis = run_analysis_agent(source, task_id) run_id = dispatch_gha(repo, analysis["prompt_b64"], analysis["branch"], analysis["runner_type"], task_id) result = wait_for_gha(repo, run_id) post_result(source, task_id, result) ``` ### Prefect Deployment Self-hosted on Eagle (or VPS) — single Docker container: ```yaml # docker-compose.yml services: prefect-server: image: prefecthq/prefect:3-latest ports: ["4200:4200"] environment: PREFECT_SERVER_DATABASE_CONNECTION_URL: "postgresql+asyncpg://..." volumes: - prefect-data:/root/.prefect prefect-worker: image: prefecthq/prefect:3-latest command: prefect worker start --pool "local-process" environment: PREFECT_API_URL: "http://prefect-server:4200/api" ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY}" GH_TOKEN: "${GH_TOKEN}" ``` Prefect UI: `http://localhost:4200` — shows all runs, retries, failures, timing. --- ## Sentry AI Analysis → Executor [PLACEHOLDER] > ⚠️ **Placeholder** — implement after Asana direct access is restored. > Validate this section against current Sentry bot plan before implementation. Planned flow: 1. Sentry receives new crash report matching severity threshold 2. Webhook triggers Prefect flow: `sentry_crash_to_executor` 3. Analysis Agent reads: crash stack trace, affected versions, reproduction frequency, linked Sentry issues 4. If fix confidence > threshold: dispatch GHA executor with targeted fix prompt 5. If confidence low: create Asana task with AI-generated analysis, tag "обработать" for human review 6. Post Sentry comment with AI analysis + PR link (if fix attempted) ``` Sentry webhook → Prefect: sentry_crash_to_executor(issue_id) → Analysis Agent: read crash + codebase context → confidence ≥ 0.7? → dispatch GHA executor → confidence < 0.7? → create Asana task + post Sentry comment → Result: PR draft OR Asana task + Sentry analysis comment ``` --- ## Peekaboo on GHA — Revised Assessment Earlier assessment ("SIP blocks TCC") was incorrect. Actual situation: - GHA runner user has `sudo` access - SIP protects system files (`/System`, `/usr`) — NOT user-level `TCC.db` - `TCC.db` is at `~/Library/Application Support/com.apple.TCC/TCC.db` - Can be modified via `sqlite3` with sudo → grants Screen Recording + Accessibility - This is a standard CI pattern for screenshot testing Xcode automation already works on GHA (AppleScript/XCUITest) — same TCC layer. **Revised conclusion: Peekaboo on GHA hosted runner is likely viable.** Needs validation in Phase 1. Fallback if sqlite3 approach fails: ddg-vm via SSH from GHA job (existing infra, confirmed working). **Solution for visual reproduction:** ``` GHA job (macos-15): → build app artifact → SSH into ddg-vm on Eagle → upload artifact to VM → run Peekaboo-based test inside VM → retrieve screenshot + result → post to PR ``` ddg-vm is existing infra with Peekaboo pre-installed + TCC permissions granted. This is already how `macOS UI Tests CI` works today. --- ## Org-Scale Design For scaling beyond personal use to full org: ### Capabilities Registry Each repo declares its execution capabilities in `.github/executor-capabilities.json`: ```json { "runner_types": ["macos-15", "ubuntu-latest"], "has_ios_build": true, "has_ui_tests": true, "has_peekaboo_vm": true, "max_concurrent_workers": 2 } ``` Prefect's Analysis Agent reads this to select the correct runner per task. ### Shared Infrastructure | Component | Where | Notes | |---|---|---| | Prefect server | Eagle or VPS | Single instance, all repos | | `ANTHROPIC_CI_KEY` | GitHub org secrets | Shared across repos | | `ASANA_CI_TOKEN` | GitHub org secrets | Rate-limited bot token | | executor-bot (MM) | MM Admin → see Asana task | Bot account for notifications | | ddg-vm pool | Eagle | 2 VMs available concurrently | ### Multi-Repo Flows ```python @flow def route_task(task_id: str): """Routes task to correct repo + executor.""" repo = classify_repo(task_id) # Asana tags → repo mapping execute_task(source="asana", task_id=task_id, repo=repo) ``` --- ## GHA `@claude` Trigger (Direct, No Orchestration Needed) For PR-level tasks, Anthropic's action handles everything natively: ```yaml # In any repo: .github/workflows/claude.yml on: issue_comment: types: [created] pull_request_review_comment: types: [created] jobs: claude: runs-on: ubuntu-latest steps: - uses: anthropics/claude-code-action@v1 with: anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }} trigger_phrase: "@claude" ``` Usage: `@claude fix the failing unit test in this PR` → Claude reads diff, fixes, pushes commit. No Prefect needed — this path is synchronous and triggered by humans. --- ## Task Intake & Scope Control ### Asana Tagging Contract Only tasks explicitly tagged for automation enter the pipeline. The agent never self-selects tasks. | Tag | Meaning | |---|---| | `executor:ready` | Task approved for autonomous execution — analyst reviews and tags this | | `executor:analyze` | Analysis Agent runs, generates prompt, but does NOT dispatch GHA — output goes to Asana comment for human review | | `executor:hold` | Task in queue, blocked (dependency, unclear scope) | Workflow: 1. Alex (or designated reviewer) tags task `executor:ready` in Asana 2. Prefect polls Asana for tasks with this tag (every N min, or webhook) 3. Tag is removed / replaced with `executor:in-progress` once dispatched 4. No tag = not touched, ever ### Scope Guardrails (Eagle auto-approval layer) From [[executor-orchestrator-redesign]]: Eagle monitors `#executor` channel using message protocol. GHA workers use same `[REQUEST:]` / `[ESCALATE:]` protocol — Eagle is the gatekeeper regardless of where the worker runs. ``` Asana task tagged executor:ready → Prefect: run_analysis_agent() → Prefect: dispatch_gha() → GHA worker posts [STATUS:] / [REQUEST:] to MM #executor-queue → Eagle monitoring cron (2 min) reads channel → Auto-approve safe actions, escalate ambiguous to Alex → Worker proceeds only after approval ``` Auto-approval rules (same as orchestrator redesign): - `create_worktree`, `run_build`, `run_tests`, `open_draft_pr`, `push_branch` → ✅ silent approve - `post_asana_comment`, any scope expansion → ⚠️ escalate to Alex - `merge_pr` → ❌ always escalate ### Phase 1: GHA Validation (1-2 days) - [ ] Create `executor-worker.yml` in apple-browsers repo - [ ] Test `claude-code-action@v1` with trivial prompt on macOS-15 runner - [ ] Verify DerivedData cache (target: <10 min cached build) - [ ] Test ddg-vm SSH from GHA job (Peekaboo path) - [ ] Add `ANTHROPIC_CI_KEY` to org secrets ### Phase 2: Analysis Agent (2-3 days) - [ ] Write `analysis-agent.md` prompt template - [ ] Test against 3 real Asana bug tasks - [ ] Tune confidence scoring + runner selection - [ ] Validate prompt quality → PR quality correlation ### Phase 3: Prefect Orchestration (2-3 days) - [ ] Deploy Prefect server (Docker on Eagle or VPS) - [ ] Implement `execute_task` flow - [ ] Wire Asana "обработать" tag → Prefect trigger - [ ] Test end-to-end: Asana task → PR ### Phase 4: MM Integration (1 day) - [ ] Create executor-bot in MM (blocked by IT Ops — see Asana task) - [ ] Add MM notification step to Prefect flow - [ ] Create `#executor-queue` channel ### Phase 5: Sentry Integration [PLACEHOLDER] - [ ] Validate against current Sentry bot plan (requires Asana access restoration) - [ ] Implement `sentry_crash_to_executor` flow - [ ] Set confidence threshold for auto-fix vs. triage ### Phase 6: Org Rollout - [ ] Publish `executor-capabilities.json` spec - [ ] Template `executor-worker.yml` as reusable workflow - [ ] Onboard second repo (e.g. privacy-configuration) - [ ] Add `route_task` multi-repo flow to Prefect --- ## Research Findings (2026-05-14) ### GHA macOS Runners - `macos-15` = current `macos-latest` (since Sep 2025) - Xcode: max 3 simulator runtimes per image - Simulators: headless, no display server needed → XCUITests work - Build cache: DerivedData cacheable via `actions/cache` ### Peekaboo on GHA - Requires Screen Recording + Accessibility TCC permissions - SIP enabled on hosted runners → `tccutil insert` blocked - **Viable path: ddg-vm SSH from GHA job** (existing infra) ### `claude-code-action@v1` (GA) - Official Anthropic action, replaces manual `claude -p` - `workflow_dispatch` trigger works for external orchestration - `--model claude-opus-4-7` for complex fixes, sonnet for quick ones - GH token scoped to repo → no additional secrets needed for PR creation ### Prefect 3.x - Python decorators, low learning curve - Self-hosted: single Docker container + Postgres - Prefect Cloud free tier: 3 workspaces, unlimited flows (but limited runs/month) - Retry pattern for GHA polling: `@task(retries=60, retry_delay_seconds=60)` --- ## Notes - Eagle's current cron executor tick continues to run in parallel during transition - GHA `macos-15` runner: Xcode NOT pre-cached → first build ~30 min, cached ~8 min - Self-hosted runner on Eagle: avoids cache miss but ties up main Mac - DuckDuckGo has existing GHA macOS setup — reuse existing Xcode bootstrap steps - `ANTHROPIC_CI_KEY` = separate key from personal key, with usage limits