487 lines
18 KiB
Markdown
487 lines
18 KiB
Markdown
# Plan: GHA CI Executor — AI-Powered Bug Fix Pipeline
|
|
|
|
**Date**: 2026-05-14
|
|
**Status**: Planning
|
|
**Owner**: Alex M
|
|
|
|
## Related
|
|
|
|
- **Asana task: MM Bots**: https://app.asana.com/1/137249556945/project/908478224964033/task/1214799615211686
|
|
- **Mattermost Migration Plan**: [[mattermost-migration]]
|
|
- **Executor Orchestrator Redesign**: [[executor-orchestrator-redesign]]
|
|
- **Executor Orchestrator Redesign**: [[executor-orchestrator-redesign]]
|
|
|
|
---
|
|
|
|
## Goal
|
|
|
|
Build an AI-powered bug fix pipeline that scales from personal use to org-wide deployment:
|
|
|
|
- Tasks arrive from multiple sources (Asana, Sentry, `@claude` in PRs)
|
|
- Analysis Agent generates a rich, context-aware prompt
|
|
- GHA executes Claude Code on the right runner (macOS, Linux, self-hosted)
|
|
- Results posted back to source (Asana task, Sentry issue, PR comment, MM channel)
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
Three distinct layers, each independently replaceable:
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ TRIGGER LAYER │
|
|
│ │
|
|
│ Asana task tagged "обработать" │
|
|
│ Sentry crash report [→ AI Analysis Placeholder] │
|
|
│ GitHub: @claude mention in PR/Issue (native, no glue) │
|
|
│ Eagle: on-demand ("go [GID]") │
|
|
└───────────────┬─────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ ORCHESTRATION LAYER — Prefect │
|
|
│ │
|
|
│ Flow: execute_task(source, task_id) │
|
|
│ 1. Analysis Agent (Claude) → structured prompt │
|
|
│ 2. Select execution target (repo + runner type) │
|
|
│ 3. Dispatch GHA workflow via `gh workflow run` │
|
|
│ 4. Monitor job status (poll / webhook) │
|
|
│ 5. On complete: post result to source + notify MM │
|
|
│ │
|
|
│ State: PostgreSQL | UI: Prefect dashboard │
|
|
│ Concurrency: max N in-flight | Retry: built-in │
|
|
└───────────────┬─────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ EXECUTION LAYER — GitHub Actions │
|
|
│ │
|
|
│ uses: anthropics/claude-code-action@v1 (official GA) │
|
|
│ │
|
|
│ Runner selection by task type: │
|
|
│ ├── macOS bug (iOS/macOS): macos-15 or self-hosted Mac │
|
|
│ ├── macOS visual repro: ddg-vm (Peekaboo inside VM) │
|
|
│ └── General (Linux): ubuntu-latest │
|
|
│ │
|
|
│ Output: draft PR + result artifact (JSON) │
|
|
└───────────────┬─────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ NOTIFICATION LAYER — Mattermost / Zulip │
|
|
│ │
|
|
│ executor-bot → #executor-queue (MM) │
|
|
│ Summary: task, PR link, test result, confidence score │
|
|
└─────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Analysis Agent
|
|
|
|
Runs inside Prefect flow (on Eagle or serverless), before GHA dispatch:
|
|
|
|
**Input:** task source (Asana GID / Sentry issue ID / GitHub issue URL)
|
|
|
|
**Steps:**
|
|
1. Read full task context: description, comments, user reports, related code
|
|
2. Search codebase for relevant files (git grep, symbol search)
|
|
3. Generate structured executor prompt:
|
|
- Problem statement
|
|
- Reproduction steps
|
|
- Investigation plan (files to check, hypotheses)
|
|
- Acceptance criteria (what "fixed" looks like)
|
|
- Test strategy (unit / UI / Peekaboo visual)
|
|
4. Classify task → select runner type
|
|
|
|
**Output:** `executor_prompt.md` + runner classification + branch name
|
|
|
|
---
|
|
|
|
## Execution Layer: `claude-code-action@v1`
|
|
|
|
Anthropic's official GA action (released 2025). Key advantages over manual `claude -p`:
|
|
- Handles GitHub context natively (PR diffs, issue body, comments)
|
|
- Auto-posts results as PR review comments
|
|
- `--dangerously-skip-permissions` not needed — action has scoped GitHub token
|
|
- Works with `workflow_dispatch` for external triggers
|
|
|
|
```yaml
|
|
name: Executor Worker
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
task_gid:
|
|
description: 'Asana task GID'
|
|
required: true
|
|
executor_prompt:
|
|
description: 'Base64-encoded executor prompt'
|
|
required: true
|
|
branch_name:
|
|
description: 'Git branch to create'
|
|
required: true
|
|
runner_type:
|
|
description: 'macos-15 | ubuntu-latest | self-hosted'
|
|
default: 'macos-15'
|
|
|
|
jobs:
|
|
executor:
|
|
runs-on: ${{ inputs.runner_type }}
|
|
timeout-minutes: 120
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: main
|
|
fetch-depth: 0
|
|
|
|
- name: Decode prompt
|
|
run: echo "${{ inputs.executor_prompt }}" | base64 -d > /tmp/executor-prompt.md
|
|
|
|
- name: Run Claude Code
|
|
uses: anthropics/claude-code-action@v1
|
|
with:
|
|
prompt: /tmp/executor-prompt.md
|
|
anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }}
|
|
github_token: ${{ secrets.GITHUB_TOKEN }}
|
|
claude_args: |
|
|
--model claude-opus-4-7
|
|
--max-turns 30
|
|
|
|
- name: Post result to Asana
|
|
if: always()
|
|
env:
|
|
ASANA_TOKEN: ${{ secrets.ASANA_CI_TOKEN }}
|
|
run: |
|
|
# Post PR link + test summary to Asana task ${{ inputs.task_gid }}
|
|
# See scripts/post-executor-result.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Orchestration Layer: Prefect
|
|
|
|
### Why Prefect (not Temporal, not Kestra)
|
|
|
|
| | Prefect | Temporal | Kestra |
|
|
|---|---|---|---|
|
|
| Learning curve | Low (Python decorators) | Very high (~1 month) | Medium (YAML) |
|
|
| AI/agent workflows | ✅ Native Python | ⚠️ Lots of boilerplate | ❌ YAML gets messy |
|
|
| Self-hosted | ✅ Docker, simple | ✅ Complex cluster | ✅ Docker |
|
|
| Retry / state | ✅ Built-in | ✅ Indestructible | ✅ Kafka-backed |
|
|
| Org-scale RBAC | Prefect Cloud (paid) | ✅ | ✅ |
|
|
| **Fit for this use case** | **✅ Best** | ⚠️ Overkill | ⚠️ Wrong paradigm |
|
|
|
|
Temporal is overkill: our tasks complete in <2h, don't need month-long replay guarantees.
|
|
Kestra is designed for ETL/data, not AI agent orchestration.
|
|
|
|
### Prefect Flow Design
|
|
|
|
```python
|
|
from prefect import flow, task
|
|
import subprocess, base64, time
|
|
|
|
@task(retries=2, retry_delay_seconds=60)
|
|
def run_analysis_agent(task_source: str, task_id: str) -> dict:
|
|
"""Claude analyzes the task and returns structured executor prompt."""
|
|
# Runs Claude (hermes or claude -p) with task context
|
|
# Returns: {prompt_b64, branch_name, runner_type, confidence}
|
|
|
|
@task
|
|
def dispatch_gha(repo: str, prompt_b64: str, branch: str, runner: str, task_gid: str) -> str:
|
|
"""Triggers GHA workflow_dispatch. Returns run_id."""
|
|
result = subprocess.run([
|
|
"gh", "workflow", "run", "executor-worker.yml",
|
|
"--repo", repo,
|
|
"-f", f"executor_prompt={prompt_b64}",
|
|
"-f", f"branch_name={branch}",
|
|
"-f", f"runner_type={runner}",
|
|
"-f", f"task_gid={task_gid}",
|
|
], capture_output=True, text=True)
|
|
return extract_run_id(result.stdout)
|
|
|
|
@task(retries=60, retry_delay_seconds=60)
|
|
def wait_for_gha(repo: str, run_id: str) -> dict:
|
|
"""Polls GHA job until complete. Returns result artifact."""
|
|
status = get_gha_status(repo, run_id)
|
|
if status in ("in_progress", "queued"):
|
|
raise Exception("Still running") # triggers retry
|
|
return get_result_artifact(repo, run_id)
|
|
|
|
@task
|
|
def post_result(source: str, task_id: str, result: dict):
|
|
"""Posts PR link + summary back to Asana/Sentry/MM."""
|
|
...
|
|
|
|
@flow(name="executor-task")
|
|
def execute_task(source: str, task_id: str, repo: str = "duckduckgo/apple-browsers"):
|
|
analysis = run_analysis_agent(source, task_id)
|
|
run_id = dispatch_gha(repo, analysis["prompt_b64"], analysis["branch"], analysis["runner_type"], task_id)
|
|
result = wait_for_gha(repo, run_id)
|
|
post_result(source, task_id, result)
|
|
```
|
|
|
|
### Prefect Deployment
|
|
|
|
Self-hosted on Eagle (or VPS) — single Docker container:
|
|
|
|
```yaml
|
|
# docker-compose.yml
|
|
services:
|
|
prefect-server:
|
|
image: prefecthq/prefect:3-latest
|
|
ports: ["4200:4200"]
|
|
environment:
|
|
PREFECT_SERVER_DATABASE_CONNECTION_URL: "postgresql+asyncpg://..."
|
|
volumes:
|
|
- prefect-data:/root/.prefect
|
|
|
|
prefect-worker:
|
|
image: prefecthq/prefect:3-latest
|
|
command: prefect worker start --pool "local-process"
|
|
environment:
|
|
PREFECT_API_URL: "http://prefect-server:4200/api"
|
|
ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY}"
|
|
GH_TOKEN: "${GH_TOKEN}"
|
|
```
|
|
|
|
Prefect UI: `http://localhost:4200` — shows all runs, retries, failures, timing.
|
|
|
|
---
|
|
|
|
## Sentry AI Analysis → Executor [PLACEHOLDER]
|
|
|
|
> ⚠️ **Placeholder** — implement after Asana direct access is restored.
|
|
> Validate this section against current Sentry bot plan before implementation.
|
|
|
|
Planned flow:
|
|
1. Sentry receives new crash report matching severity threshold
|
|
2. Webhook triggers Prefect flow: `sentry_crash_to_executor`
|
|
3. Analysis Agent reads: crash stack trace, affected versions, reproduction frequency, linked Sentry issues
|
|
4. If fix confidence > threshold: dispatch GHA executor with targeted fix prompt
|
|
5. If confidence low: create Asana task with AI-generated analysis, tag "обработать" for human review
|
|
6. Post Sentry comment with AI analysis + PR link (if fix attempted)
|
|
|
|
```
|
|
Sentry webhook
|
|
→ Prefect: sentry_crash_to_executor(issue_id)
|
|
→ Analysis Agent: read crash + codebase context
|
|
→ confidence ≥ 0.7? → dispatch GHA executor
|
|
→ confidence < 0.7? → create Asana task + post Sentry comment
|
|
→ Result: PR draft OR Asana task + Sentry analysis comment
|
|
```
|
|
|
|
---
|
|
|
|
## Peekaboo on GHA — Revised Assessment
|
|
|
|
Earlier assessment ("SIP blocks TCC") was incorrect. Actual situation:
|
|
|
|
- GHA runner user has `sudo` access
|
|
- SIP protects system files (`/System`, `/usr`) — NOT user-level `TCC.db`
|
|
- `TCC.db` is at `~/Library/Application Support/com.apple.TCC/TCC.db`
|
|
- Can be modified via `sqlite3` with sudo → grants Screen Recording + Accessibility
|
|
- This is a standard CI pattern for screenshot testing
|
|
|
|
Xcode automation already works on GHA (AppleScript/XCUITest) — same TCC layer.
|
|
|
|
**Revised conclusion: Peekaboo on GHA hosted runner is likely viable.** Needs validation in Phase 1.
|
|
|
|
Fallback if sqlite3 approach fails: ddg-vm via SSH from GHA job (existing infra, confirmed working).
|
|
|
|
**Solution for visual reproduction:**
|
|
|
|
```
|
|
GHA job (macos-15):
|
|
→ build app artifact
|
|
→ SSH into ddg-vm on Eagle
|
|
→ upload artifact to VM
|
|
→ run Peekaboo-based test inside VM
|
|
→ retrieve screenshot + result
|
|
→ post to PR
|
|
```
|
|
|
|
ddg-vm is existing infra with Peekaboo pre-installed + TCC permissions granted.
|
|
This is already how `macOS UI Tests CI` works today.
|
|
|
|
---
|
|
|
|
## Org-Scale Design
|
|
|
|
For scaling beyond personal use to full org:
|
|
|
|
### Capabilities Registry
|
|
|
|
Each repo declares its execution capabilities in `.github/executor-capabilities.json`:
|
|
```json
|
|
{
|
|
"runner_types": ["macos-15", "ubuntu-latest"],
|
|
"has_ios_build": true,
|
|
"has_ui_tests": true,
|
|
"has_peekaboo_vm": true,
|
|
"max_concurrent_workers": 2
|
|
}
|
|
```
|
|
|
|
Prefect's Analysis Agent reads this to select the correct runner per task.
|
|
|
|
### Shared Infrastructure
|
|
|
|
| Component | Where | Notes |
|
|
|---|---|---|
|
|
| Prefect server | Eagle or VPS | Single instance, all repos |
|
|
| `ANTHROPIC_CI_KEY` | GitHub org secrets | Shared across repos |
|
|
| `ASANA_CI_TOKEN` | GitHub org secrets | Rate-limited bot token |
|
|
| executor-bot (MM) | MM Admin → see Asana task | Bot account for notifications |
|
|
| ddg-vm pool | Eagle | 2 VMs available concurrently |
|
|
|
|
### Multi-Repo Flows
|
|
|
|
```python
|
|
@flow
|
|
def route_task(task_id: str):
|
|
"""Routes task to correct repo + executor."""
|
|
repo = classify_repo(task_id) # Asana tags → repo mapping
|
|
execute_task(source="asana", task_id=task_id, repo=repo)
|
|
```
|
|
|
|
---
|
|
|
|
## GHA `@claude` Trigger (Direct, No Orchestration Needed)
|
|
|
|
For PR-level tasks, Anthropic's action handles everything natively:
|
|
|
|
```yaml
|
|
# In any repo: .github/workflows/claude.yml
|
|
on:
|
|
issue_comment:
|
|
types: [created]
|
|
pull_request_review_comment:
|
|
types: [created]
|
|
jobs:
|
|
claude:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: anthropics/claude-code-action@v1
|
|
with:
|
|
anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }}
|
|
trigger_phrase: "@claude"
|
|
```
|
|
|
|
Usage: `@claude fix the failing unit test in this PR` → Claude reads diff, fixes, pushes commit.
|
|
No Prefect needed — this path is synchronous and triggered by humans.
|
|
|
|
---
|
|
|
|
## Task Intake & Scope Control
|
|
|
|
### Asana Tagging Contract
|
|
|
|
Only tasks explicitly tagged for automation enter the pipeline. The agent never self-selects tasks.
|
|
|
|
| Tag | Meaning |
|
|
|---|---|
|
|
| `executor:ready` | Task approved for autonomous execution — analyst reviews and tags this |
|
|
| `executor:analyze` | Analysis Agent runs, generates prompt, but does NOT dispatch GHA — output goes to Asana comment for human review |
|
|
| `executor:hold` | Task in queue, blocked (dependency, unclear scope) |
|
|
|
|
Workflow:
|
|
1. Alex (or designated reviewer) tags task `executor:ready` in Asana
|
|
2. Prefect polls Asana for tasks with this tag (every N min, or webhook)
|
|
3. Tag is removed / replaced with `executor:in-progress` once dispatched
|
|
4. No tag = not touched, ever
|
|
|
|
### Scope Guardrails (Eagle auto-approval layer)
|
|
|
|
From [[executor-orchestrator-redesign]]: Eagle monitors `#executor` channel using message protocol.
|
|
GHA workers use same `[REQUEST:]` / `[ESCALATE:]` protocol — Eagle is the gatekeeper regardless of where the worker runs.
|
|
|
|
```
|
|
Asana task tagged executor:ready
|
|
→ Prefect: run_analysis_agent()
|
|
→ Prefect: dispatch_gha()
|
|
→ GHA worker posts [STATUS:] / [REQUEST:] to MM #executor-queue
|
|
→ Eagle monitoring cron (2 min) reads channel
|
|
→ Auto-approve safe actions, escalate ambiguous to Alex
|
|
→ Worker proceeds only after approval
|
|
```
|
|
|
|
Auto-approval rules (same as orchestrator redesign):
|
|
- `create_worktree`, `run_build`, `run_tests`, `open_draft_pr`, `push_branch` → ✅ silent approve
|
|
- `post_asana_comment`, any scope expansion → ⚠️ escalate to Alex
|
|
- `merge_pr` → ❌ always escalate
|
|
|
|
|
|
|
|
### Phase 1: GHA Validation (1-2 days)
|
|
- [ ] Create `executor-worker.yml` in apple-browsers repo
|
|
- [ ] Test `claude-code-action@v1` with trivial prompt on macOS-15 runner
|
|
- [ ] Verify DerivedData cache (target: <10 min cached build)
|
|
- [ ] Test ddg-vm SSH from GHA job (Peekaboo path)
|
|
- [ ] Add `ANTHROPIC_CI_KEY` to org secrets
|
|
|
|
### Phase 2: Analysis Agent (2-3 days)
|
|
- [ ] Write `analysis-agent.md` prompt template
|
|
- [ ] Test against 3 real Asana bug tasks
|
|
- [ ] Tune confidence scoring + runner selection
|
|
- [ ] Validate prompt quality → PR quality correlation
|
|
|
|
### Phase 3: Prefect Orchestration (2-3 days)
|
|
- [ ] Deploy Prefect server (Docker on Eagle or VPS)
|
|
- [ ] Implement `execute_task` flow
|
|
- [ ] Wire Asana "обработать" tag → Prefect trigger
|
|
- [ ] Test end-to-end: Asana task → PR
|
|
|
|
### Phase 4: MM Integration (1 day)
|
|
- [ ] Create executor-bot in MM (blocked by IT Ops — see Asana task)
|
|
- [ ] Add MM notification step to Prefect flow
|
|
- [ ] Create `#executor-queue` channel
|
|
|
|
### Phase 5: Sentry Integration [PLACEHOLDER]
|
|
- [ ] Validate against current Sentry bot plan (requires Asana access restoration)
|
|
- [ ] Implement `sentry_crash_to_executor` flow
|
|
- [ ] Set confidence threshold for auto-fix vs. triage
|
|
|
|
### Phase 6: Org Rollout
|
|
- [ ] Publish `executor-capabilities.json` spec
|
|
- [ ] Template `executor-worker.yml` as reusable workflow
|
|
- [ ] Onboard second repo (e.g. privacy-configuration)
|
|
- [ ] Add `route_task` multi-repo flow to Prefect
|
|
|
|
---
|
|
|
|
## Research Findings (2026-05-14)
|
|
|
|
### GHA macOS Runners
|
|
- `macos-15` = current `macos-latest` (since Sep 2025)
|
|
- Xcode: max 3 simulator runtimes per image
|
|
- Simulators: headless, no display server needed → XCUITests work
|
|
- Build cache: DerivedData cacheable via `actions/cache`
|
|
|
|
### Peekaboo on GHA
|
|
- Requires Screen Recording + Accessibility TCC permissions
|
|
- SIP enabled on hosted runners → `tccutil insert` blocked
|
|
- **Viable path: ddg-vm SSH from GHA job** (existing infra)
|
|
|
|
### `claude-code-action@v1` (GA)
|
|
- Official Anthropic action, replaces manual `claude -p`
|
|
- `workflow_dispatch` trigger works for external orchestration
|
|
- `--model claude-opus-4-7` for complex fixes, sonnet for quick ones
|
|
- GH token scoped to repo → no additional secrets needed for PR creation
|
|
|
|
### Prefect 3.x
|
|
- Python decorators, low learning curve
|
|
- Self-hosted: single Docker container + Postgres
|
|
- Prefect Cloud free tier: 3 workspaces, unlimited flows (but limited runs/month)
|
|
- Retry pattern for GHA polling: `@task(retries=60, retry_delay_seconds=60)`
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- Eagle's current cron executor tick continues to run in parallel during transition
|
|
- GHA `macos-15` runner: Xcode NOT pre-cached → first build ~30 min, cached ~8 min
|
|
- Self-hosted runner on Eagle: avoids cache miss but ties up main Mac
|
|
- DuckDuckGo has existing GHA macOS setup — reuse existing Xcode bootstrap steps
|
|
- `ANTHROPIC_CI_KEY` = separate key from personal key, with usage limits
|