From ed535b986c3e681b18d8c216eecb4e6bd608f5d3 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Thu, 14 May 2026 14:12:44 +0600 Subject: [PATCH] =?UTF-8?q?[2026-05-14]=20gha-ci-executor:=20major=20plan?= =?UTF-8?q?=20revision=20=E2=80=94=20Prefect=20orchestration,=20claude-cod?= =?UTF-8?q?e-action@v1,=20org-scale=20design,=20Sentry=20placeholder?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- work/plans/gha-ci-executor.md | 503 ++++++++++++++++++++++++---------- 1 file changed, 362 insertions(+), 141 deletions(-) diff --git a/work/plans/gha-ci-executor.md b/work/plans/gha-ci-executor.md index 18b60b9e..3167178b 100644 --- a/work/plans/gha-ci-executor.md +++ b/work/plans/gha-ci-executor.md @@ -1,42 +1,109 @@ -# Plan: GHA CI Executor — macOS Build + Peekaboo +# Plan: GHA CI Executor — AI-Powered Bug Fix Pipeline **Date**: 2026-05-14 **Status**: Planning +**Owner**: Alex M + +## Related + +- **Asana task: MM Bots**: https://app.asana.com/1/137249556945/project/908478224964033/task/1214799615211686 +- **Mattermost Migration Plan**: [[mattermost-migration]] + +--- ## Goal -Migrate executor workers from Eagle's local Mac to GitHub Actions macOS runners: -- Each task spawns a GHA job -- Job installs Claude Code + Peekaboo + builds the app -- Executor (Claude Code) runs the fix, opens PR, posts report to Asana -- Eagle orchestrates: queues task → triggers GHA → monitors job → handles result +Build an AI-powered bug fix pipeline that scales from personal use to org-wide deployment: + +- Tasks arrive from multiple sources (Asana, Sentry, `@claude` in PRs) +- Analysis Agent generates a rich, context-aware prompt +- GHA executes Claude Code on the right runner (macOS, Linux, self-hosted) +- Results posted back to source (Asana task, Sentry issue, PR comment, MM channel) + +--- ## Architecture +Three distinct layers, each independently replaceable: + ``` -Eagle (orchestrator) → triggers GHA workflow via gh CLI - ↓ - GHA macOS runner (macos-15-xlarge or self-hosted) - ↓ - Install: Xcode, Claude Code, Peekaboo, Simulator - ↓ - claude -p executor-worker.md (non-interactive) - ↓ - Claude Code: fix → build → UI test via Peekaboo → PR - ↓ - Post result to Asana (gh pr comment + asana-bot) +┌─────────────────────────────────────────────────────────┐ +│ TRIGGER LAYER │ +│ │ +│ Asana task tagged "обработать" │ +│ Sentry crash report [→ AI Analysis Placeholder] │ +│ GitHub: @claude mention in PR/Issue (native, no glue) │ +│ Eagle: on-demand ("go [GID]") │ +└───────────────┬─────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ ORCHESTRATION LAYER — Prefect │ +│ │ +│ Flow: execute_task(source, task_id) │ +│ 1. Analysis Agent (Claude) → structured prompt │ +│ 2. Select execution target (repo + runner type) │ +│ 3. Dispatch GHA workflow via `gh workflow run` │ +│ 4. Monitor job status (poll / webhook) │ +│ 5. On complete: post result to source + notify MM │ +│ │ +│ State: PostgreSQL | UI: Prefect dashboard │ +│ Concurrency: max N in-flight | Retry: built-in │ +└───────────────┬─────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ EXECUTION LAYER — GitHub Actions │ +│ │ +│ uses: anthropics/claude-code-action@v1 (official GA) │ +│ │ +│ Runner selection by task type: │ +│ ├── macOS bug (iOS/macOS): macos-15 or self-hosted Mac │ +│ ├── macOS visual repro: ddg-vm (Peekaboo inside VM) │ +│ └── General (Linux): ubuntu-latest │ +│ │ +│ Output: draft PR + result artifact (JSON) │ +└───────────────┬─────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ NOTIFICATION LAYER — Mattermost / Zulip │ +│ │ +│ executor-bot → #executor-queue (MM) │ +│ Summary: task, PR link, test result, confidence score │ +└─────────────────────────────────────────────────────────┘ ``` -## Analysis Agent (Cloud-side) +--- -Separate from the executor worker: -- Runs in Claude Cloud (or Eagle's Mac) -- Picks task tagged "обработать" from Asana -- Reads full task context (description, stories, user reports) -- Generates detailed executor prompt (investigation plan + acceptance criteria) -- Triggers GHA workflow with this prompt as input +## Analysis Agent -## GHA Workflow Design +Runs inside Prefect flow (on Eagle or serverless), before GHA dispatch: + +**Input:** task source (Asana GID / Sentry issue ID / GitHub issue URL) + +**Steps:** +1. Read full task context: description, comments, user reports, related code +2. Search codebase for relevant files (git grep, symbol search) +3. Generate structured executor prompt: + - Problem statement + - Reproduction steps + - Investigation plan (files to check, hypotheses) + - Acceptance criteria (what "fixed" looks like) + - Test strategy (unit / UI / Peekaboo visual) +4. Classify task → select runner type + +**Output:** `executor_prompt.md` + runner classification + branch name + +--- + +## Execution Layer: `claude-code-action@v1` + +Anthropic's official GA action (released 2025). Key advantages over manual `claude -p`: +- Handles GitHub context natively (PR diffs, issue body, comments) +- Auto-posts results as PR review comments +- `--dangerously-skip-permissions` not needed — action has scoped GitHub token +- Works with `workflow_dispatch` for external triggers ```yaml name: Executor Worker @@ -45,169 +112,323 @@ on: inputs: task_gid: description: 'Asana task GID' + required: true executor_prompt: description: 'Base64-encoded executor prompt' + required: true branch_name: description: 'Git branch to create' + required: true + runner_type: + description: 'macos-15 | ubuntu-latest | self-hosted' + default: 'macos-15' jobs: executor: - runs-on: macos-15 # or self-hosted M-series + runs-on: ${{ inputs.runner_type }} timeout-minutes: 120 steps: - uses: actions/checkout@v4 - - name: Install Claude Code - run: npm install -g @anthropic-ai/claude-code - - name: Install Peekaboo - run: # brew install or download binary - - name: Bootstrap Xcode - run: # select Xcode version, accept license - - name: Run Executor - env: - ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_CI_KEY }} - ASANA_TOKEN: ${{ secrets.ASANA_CI_TOKEN }} - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: | - echo "${{ inputs.executor_prompt }}" | base64 -d > /tmp/prompt.md - claude -p /tmp/prompt.md \ - --branch "${{ inputs.branch_name }}" \ - --no-interactive - - name: Post Result + with: + ref: main + fetch-depth: 0 + + - name: Decode prompt + run: echo "${{ inputs.executor_prompt }}" | base64 -d > /tmp/executor-prompt.md + + - name: Run Claude Code + uses: anthropics/claude-code-action@v1 + with: + prompt: /tmp/executor-prompt.md + anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }} + github_token: ${{ secrets.GITHUB_TOKEN }} + claude_args: | + --model claude-opus-4-7 + --max-turns 30 + + - name: Post result to Asana if: always() - run: # Post summary to Asana task + open PR if not already + env: + ASANA_TOKEN: ${{ secrets.ASANA_CI_TOKEN }} + run: | + # Post PR link + test summary to Asana task ${{ inputs.task_gid }} + # See scripts/post-executor-result.sh ``` -## Research Findings (2026-05-14) +--- -### GHA macOS Runners — Current State +## Orchestration Layer: Prefect -- `macos-15` is current `macos-latest` (since Sep 2025) -- Xcode max 3 simulator runtimes per image (since Aug 11, 2025) -- **Simulators work fine on GHA** — iOS Simulator runs headless without display server -- XCUITests run via `xcodebuild test` on Simulator → headless OK, no Screen Recording needed +### Why Prefect (not Temporal, not Kestra) -### Peekaboo on GHA — The Problem +| | Prefect | Temporal | Kestra | +|---|---|---|---| +| Learning curve | Low (Python decorators) | Very high (~1 month) | Medium (YAML) | +| AI/agent workflows | ✅ Native Python | ⚠️ Lots of boilerplate | ❌ YAML gets messy | +| Self-hosted | ✅ Docker, simple | ✅ Complex cluster | ✅ Docker | +| Retry / state | ✅ Built-in | ✅ Indestructible | ✅ Kafka-backed | +| Org-scale RBAC | Prefect Cloud (paid) | ✅ | ✅ | +| **Fit for this use case** | **✅ Best** | ⚠️ Overkill | ⚠️ Wrong paradigm | -Peekaboo requires: -1. **Screen Recording** permission (TCC) — for `ScreenCaptureKit` -2. **Accessibility** permission (TCC) — for AX tree traversal -3. Runs on **macOS 15+ (Sequoia)** +Temporal is overkill: our tasks complete in <2h, don't need month-long replay guarantees. +Kestra is designed for ETL/data, not AI agent orchestration. -On GHA macOS runners: -- TCC permissions can be pre-granted via `tccutil reset` + `tccutil insert` — but requires SIP disabled or specific entitlements -- GHA macOS runners run **with SIP enabled** — `tccutil insert` won't work without sudo tricks -- **Peekaboo CLI needs the Mac App running as a bridge** (Unix socket IPC) for privileged operations +### Prefect Flow Design -**Conclusion: Peekaboo on GHA hosted runner = NOT viable out of the box.** Would need: -- Self-hosted macOS runner (Eagle itself, or a dedicated Mac) -- OR: use ddg-vm to spin a VM on Eagle, run Peekaboo inside VM (current CI pattern) +```python +from prefect import flow, task +import subprocess, base64, time -### XCUITest on GHA — Works Fine +@task(retries=2, retry_delay_seconds=60) +def run_analysis_agent(task_source: str, task_id: str) -> dict: + """Claude analyzes the task and returns structured executor prompt.""" + # Runs Claude (hermes or claude -p) with task context + # Returns: {prompt_b64, branch_name, runner_type, confidence} -Our current CI already runs `macOS UI Tests CI` scheme on GHA via ddg-vm VMs. The issue isn't XCUITest — it's **visual/Peekaboo-based** reproduction that's GHA-incompatible. +@task +def dispatch_gha(repo: str, prompt_b64: str, branch: str, runner: str, task_gid: str) -> str: + """Triggers GHA workflow_dispatch. Returns run_id.""" + result = subprocess.run([ + "gh", "workflow", "run", "executor-worker.yml", + "--repo", repo, + "-f", f"executor_prompt={prompt_b64}", + "-f", f"branch_name={branch}", + "-f", f"runner_type={runner}", + "-f", f"task_gid={task_gid}", + ], capture_output=True, text=True) + return extract_run_id(result.stdout) -### Two-Tier Approach (Recommended) +@task(retries=60, retry_delay_seconds=60) +def wait_for_gha(repo: str, run_id: str) -> dict: + """Polls GHA job until complete. Returns result artifact.""" + status = get_gha_status(repo, run_id) + if status in ("in_progress", "queued"): + raise Exception("Still running") # triggers retry + return get_result_artifact(repo, run_id) -``` -Analysis Agent (Claude Cloud / Eagle) - ↓ reads Asana task, generates prompt - ↓ triggers GHA workflow +@task +def post_result(source: str, task_id: str, result: dict): + """Posts PR link + summary back to Asana/Sentry/MM.""" + ... -GHA macOS runner: - ↓ checkout + build (cached DerivedData) - ↓ claude -p executor-worker.md (non-interactive) - ↓ fix code, run unit + UI tests via Simulator (NO Peekaboo) - ↓ open draft PR - -For visual reproduction / Peekaboo validation: - → spawn ddg-vm (existing infra) from Eagle or from GHA via SSH - → VM has Peekaboo pre-installed + permissions granted +@flow(name="executor-task") +def execute_task(source: str, task_id: str, repo: str = "duckduckgo/apple-browsers"): + analysis = run_analysis_agent(source, task_id) + run_id = dispatch_gha(repo, analysis["prompt_b64"], analysis["branch"], analysis["runner_type"], task_id) + result = wait_for_gha(repo, run_id) + post_result(source, task_id, result) ``` -### Claude Code Non-Interactive on GHA +### Prefect Deployment -`claude -p prompt.md` works non-interactively. Needs: -- `ANTHROPIC_API_KEY` in secrets -- `--dangerously-skip-permissions` flag (all tools pre-approved) or tool allowlist +Self-hosted on Eagle (or VPS) — single Docker container: -GHA secret: `ANTHROPIC_CI_KEY` — separate from user key. +```yaml +# docker-compose.yml +services: + prefect-server: + image: prefecthq/prefect:3-latest + ports: ["4200:4200"] + environment: + PREFECT_SERVER_DATABASE_CONNECTION_URL: "postgresql+asyncpg://..." + volumes: + - prefect-data:/root/.prefect + prefect-worker: + image: prefecthq/prefect:3-latest + command: prefect worker start --pool "local-process" + environment: + PREFECT_API_URL: "http://prefect-server:4200/api" + ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY}" + GH_TOKEN: "${GH_TOKEN}" +``` +Prefect UI: `http://localhost:4200` — shows all runs, retries, failures, timing. -## Validation Steps +--- -Before full implementation: -1. Test prod build on `macos-15` runner: does it succeed without signing? -2. Test Peekaboo install + basic screenshot on GHA runner -3. Test `claude -p` with a simple prompt on GHA runner -4. Verify DerivedData caching works (build time < 10 min cached) +## Sentry AI Analysis → Executor [PLACEHOLDER] + +> ⚠️ **Placeholder** — implement after Asana direct access is restored. +> Validate this section against current Sentry bot plan before implementation. + +Planned flow: +1. Sentry receives new crash report matching severity threshold +2. Webhook triggers Prefect flow: `sentry_crash_to_executor` +3. Analysis Agent reads: crash stack trace, affected versions, reproduction frequency, linked Sentry issues +4. If fix confidence > threshold: dispatch GHA executor with targeted fix prompt +5. If confidence low: create Asana task with AI-generated analysis, tag "обработать" for human review +6. Post Sentry comment with AI analysis + PR link (if fix attempted) + +``` +Sentry webhook + → Prefect: sentry_crash_to_executor(issue_id) + → Analysis Agent: read crash + codebase context + → confidence ≥ 0.7? → dispatch GHA executor + → confidence < 0.7? → create Asana task + post Sentry comment + → Result: PR draft OR Asana task + Sentry analysis comment +``` + +--- + +## Peekaboo on GHA — Constraint + +GHA hosted runners run with **SIP enabled** — `tccutil insert` for Screen Recording / Accessibility doesn't work. + +**Solution for visual reproduction:** + +``` +GHA job (macos-15): + → build app artifact + → SSH into ddg-vm on Eagle + → upload artifact to VM + → run Peekaboo-based test inside VM + → retrieve screenshot + result + → post to PR +``` + +ddg-vm is existing infra with Peekaboo pre-installed + TCC permissions granted. +This is already how `macOS UI Tests CI` works today. + +--- + +## Org-Scale Design + +For scaling beyond personal use to full org: + +### Capabilities Registry + +Each repo declares its execution capabilities in `.github/executor-capabilities.json`: +```json +{ + "runner_types": ["macos-15", "ubuntu-latest"], + "has_ios_build": true, + "has_ui_tests": true, + "has_peekaboo_vm": true, + "max_concurrent_workers": 2 +} +``` + +Prefect's Analysis Agent reads this to select the correct runner per task. + +### Shared Infrastructure + +| Component | Where | Notes | +|---|---|---| +| Prefect server | Eagle or VPS | Single instance, all repos | +| `ANTHROPIC_CI_KEY` | GitHub org secrets | Shared across repos | +| `ASANA_CI_TOKEN` | GitHub org secrets | Rate-limited bot token | +| executor-bot (MM) | MM Admin → see Asana task | Bot account for notifications | +| ddg-vm pool | Eagle | 2 VMs available concurrently | + +### Multi-Repo Flows + +```python +@flow +def route_task(task_id: str): + """Routes task to correct repo + executor.""" + repo = classify_repo(task_id) # Asana tags → repo mapping + execute_task(source="asana", task_id=task_id, repo=repo) +``` + +--- + +## GHA `@claude` Trigger (Direct, No Orchestration Needed) + +For PR-level tasks, Anthropic's action handles everything natively: + +```yaml +# In any repo: .github/workflows/claude.yml +on: + issue_comment: + types: [created] + pull_request_review_comment: + types: [created] +jobs: + claude: + runs-on: ubuntu-latest + steps: + - uses: anthropics/claude-code-action@v1 + with: + anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }} + trigger_phrase: "@claude" +``` + +Usage: `@claude fix the failing unit test in this PR` → Claude reads diff, fixes, pushes commit. +No Prefect needed — this path is synchronous and triggered by humans. + +--- ## Phased Implementation -### Phase 1: Validation (1-2 days) -- Create a test GHA workflow that: checks out repo, builds, runs basic UI test -- Verify Peekaboo accessibility permissions on headless runner -- Document what works / what doesn't +### Phase 1: GHA Validation (1-2 days) +- [ ] Create `executor-worker.yml` in apple-browsers repo +- [ ] Test `claude-code-action@v1` with trivial prompt on macOS-15 runner +- [ ] Verify DerivedData cache (target: <10 min cached build) +- [ ] Test ddg-vm SSH from GHA job (Peekaboo path) +- [ ] Add `ANTHROPIC_CI_KEY` to org secrets -### Phase 2: Claude Code Integration (2-3 days) -- Add Claude Code step to workflow -- Test with a trivial fix prompt -- Verify PR creation from GHA bot context +### Phase 2: Analysis Agent (2-3 days) +- [ ] Write `analysis-agent.md` prompt template +- [ ] Test against 3 real Asana bug tasks +- [ ] Tune confidence scoring + runner selection +- [ ] Validate prompt quality → PR quality correlation -### Phase 3: Analysis Agent (3-5 days) -- Cloud-side agent that picks Asana tasks + generates prompts -- Triggers GHA workflow via `gh workflow run` -- Monitors job, posts result to Asana +### Phase 3: Prefect Orchestration (2-3 days) +- [ ] Deploy Prefect server (Docker on Eagle or VPS) +- [ ] Implement `execute_task` flow +- [ ] Wire Asana "обработать" tag → Prefect trigger +- [ ] Test end-to-end: Asana task → PR -### Phase 4: Eagle Integration (1 day) -- Eagle can trigger Phase 3 analysis agent on demand -- `#executor` topic shows GHA job status -- Auto-approval for GHA-spawned actions +### Phase 4: MM Integration (1 day) +- [ ] Create executor-bot in MM (blocked by IT Ops — see Asana task) +- [ ] Add MM notification step to Prefect flow +- [ ] Create `#executor-queue` channel -## Relation to Sentry Bot +### Phase 5: Sentry Integration [PLACEHOLDER] +- [ ] Validate against current Sentry bot plan (requires Asana access restoration) +- [ ] Implement `sentry_crash_to_executor` flow +- [ ] Set confidence threshold for auto-fix vs. triage -Sentry bot pattern (already working): -- Cloud agent monitors Sentry errors -- Generates structured report -- Posts to Asana/Slack +### Phase 6: Org Rollout +- [ ] Publish `executor-capabilities.json` spec +- [ ] Template `executor-worker.yml` as reusable workflow +- [ ] Onboard second repo (e.g. privacy-configuration) +- [ ] Add `route_task` multi-repo flow to Prefect -GHA executor follows same pattern: -- Cloud agent picks Asana bug tasks -- Generates fix prompt -- GHA runner executes fix + validation +--- -## Related +## Research Findings (2026-05-14) -- **Asana task: MM Bots**: https://app.asana.com/1/137249556945/project/908478224964033/task/1214799615211686 -- **Mattermost Migration Plan**: [[mattermost-migration]] +### GHA macOS Runners +- `macos-15` = current `macos-latest` (since Sep 2025) +- Xcode: max 3 simulator runtimes per image +- Simulators: headless, no display server needed → XCUITests work +- Build cache: DerivedData cacheable via `actions/cache` -## Prefect — Orchestration Layer (Jenkins for Claude Code) +### Peekaboo on GHA +- Requires Screen Recording + Accessibility TCC permissions +- SIP enabled on hosted runners → `tccutil insert` blocked +- **Viable path: ddg-vm SSH from GHA job** (existing infra) -[Prefect](https://www.prefect.io/) as the workflow orchestration layer for GHA CI executor: +### `claude-code-action@v1` (GA) +- Official Anthropic action, replaces manual `claude -p` +- `workflow_dispatch` trigger works for external orchestration +- `--model claude-opus-4-7` for complex fixes, sonnet for quick ones +- GH token scoped to repo → no additional secrets needed for PR creation -- Prefect manages flow scheduling, retries, observability, and state persistence -- Each executor task = a Prefect flow: Analysis Agent → GHA trigger → monitor → result -- Eagle submits tasks to Prefect; Prefect handles retry logic, parallel runs, timeouts -- Prefect UI gives visibility into all executor runs (instead of Zulip-only status) -- Can replace or wrap the current cron-based executor tick -- Self-hosted Prefect server on Eagle or VPS (or Prefect Cloud free tier) +### Prefect 3.x +- Python decorators, low learning curve +- Self-hosted: single Docker container + Postgres +- Prefect Cloud free tier: 3 workspaces, unlimited flows (but limited runs/month) +- Retry pattern for GHA polling: `@task(retries=60, retry_delay_seconds=60)` -**How it fits:** -``` -Eagle / Asana → Prefect flow submitted - ↓ - Prefect orchestrates: - 1. Analysis agent (Claude): read task, generate prompt - 2. Trigger GHA workflow via gh CLI - 3. Monitor GHA job status (poll or webhook) - 4. On complete: post result to Asana, notify Eagle - ↓ - Retry on failure, log all state -``` +--- ## Notes -- GHA macos-15 runners do NOT have Xcode pre-cached — each run installs from scratch -- Self-hosted runner on Eagle avoids this but ties up the main Mac -- DuckDuckGo likely has existing GHA macOS setup — check existing workflows first -- Peekaboo may need `tccutil` or System Preferences pre-grant on runner +- Eagle's current cron executor tick continues to run in parallel during transition +- GHA `macos-15` runner: Xcode NOT pre-cached → first build ~30 min, cached ~8 min +- Self-hosted runner on Eagle: avoids cache miss but ties up main Mac +- DuckDuckGo has existing GHA macOS setup — reuse existing Xcode bootstrap steps +- `ANTHROPIC_CI_KEY` = separate key from personal key, with usage limits