git_admin/obsidian-vault
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ed535b986c3e681b18d8c216eecb4e6bd608f5d3
obsidian-vault/work/plans/gha-ci-executor.md
T

16 KiB
Raw Blame History

Plan: GHA CI Executor — AI-Powered Bug Fix Pipeline

Date: 2026-05-14 Status: Planning Owner: Alex M

Related

Goal

Build an AI-powered bug fix pipeline that scales from personal use to org-wide deployment:

  • Tasks arrive from multiple sources (Asana, Sentry, @claude in PRs)
  • Analysis Agent generates a rich, context-aware prompt
  • GHA executes Claude Code on the right runner (macOS, Linux, self-hosted)
  • Results posted back to source (Asana task, Sentry issue, PR comment, MM channel)

Architecture

Three distinct layers, each independently replaceable:

┌─────────────────────────────────────────────────────────┐
│  TRIGGER LAYER                                           │
│                                                          │
│  Asana task tagged "обработать"                         │
│  Sentry crash report  [→ AI Analysis Placeholder]       │
│  GitHub: @claude mention in PR/Issue  (native, no glue) │
│  Eagle: on-demand ("go [GID]")                          │
└───────────────┬─────────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────────┐
│  ORCHESTRATION LAYER — Prefect                           │
│                                                          │
│  Flow: execute_task(source, task_id)                     │
│    1. Analysis Agent (Claude) → structured prompt        │
│    2. Select execution target (repo + runner type)       │
│    3. Dispatch GHA workflow via `gh workflow run`         │
│    4. Monitor job status (poll / webhook)                │
│    5. On complete: post result to source + notify MM     │
│                                                          │
│  State: PostgreSQL  |  UI: Prefect dashboard             │
│  Concurrency: max N in-flight  |  Retry: built-in        │
└───────────────┬─────────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────────┐
│  EXECUTION LAYER — GitHub Actions                        │
│                                                          │
│  uses: anthropics/claude-code-action@v1  (official GA)  │
│                                                          │
│  Runner selection by task type:                          │
│  ├── macOS bug (iOS/macOS): macos-15 or self-hosted Mac  │
│  ├── macOS visual repro:    ddg-vm (Peekaboo inside VM)  │
│  └── General (Linux):       ubuntu-latest                │
│                                                          │
│  Output: draft PR + result artifact (JSON)              │
└───────────────┬─────────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────────┐
│  NOTIFICATION LAYER — Mattermost / Zulip                 │
│                                                          │
│  executor-bot → #executor-queue (MM)                     │
│  Summary: task, PR link, test result, confidence score   │
└─────────────────────────────────────────────────────────┘

Analysis Agent

Runs inside Prefect flow (on Eagle or serverless), before GHA dispatch:

Input: task source (Asana GID / Sentry issue ID / GitHub issue URL)

Steps:

  1. Read full task context: description, comments, user reports, related code
  2. Search codebase for relevant files (git grep, symbol search)
  3. Generate structured executor prompt:
    • Problem statement
    • Reproduction steps
    • Investigation plan (files to check, hypotheses)
    • Acceptance criteria (what "fixed" looks like)
    • Test strategy (unit / UI / Peekaboo visual)
  4. Classify task → select runner type

Output: executor_prompt.md + runner classification + branch name

Execution Layer: claude-code-action@v1

Anthropic's official GA action (released 2025). Key advantages over manual claude -p:

  • Handles GitHub context natively (PR diffs, issue body, comments)
  • Auto-posts results as PR review comments
  • --dangerously-skip-permissions not needed — action has scoped GitHub token
  • Works with workflow_dispatch for external triggers
name: Executor Worker
on:
  workflow_dispatch:
    inputs:
      task_gid:
        description: 'Asana task GID'
        required: true
      executor_prompt:
        description: 'Base64-encoded executor prompt'
        required: true
      branch_name:
        description: 'Git branch to create'
        required: true
      runner_type:
        description: 'macos-15 | ubuntu-latest | self-hosted'
        default: 'macos-15'

jobs:
  executor:
    runs-on: ${{ inputs.runner_type }}
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4
        with:
          ref: main
          fetch-depth: 0

      - name: Decode prompt
        run: echo "${{ inputs.executor_prompt }}" | base64 -d > /tmp/executor-prompt.md

      - name: Run Claude Code
        uses: anthropics/claude-code-action@v1
        with:
          prompt: /tmp/executor-prompt.md
          anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }}
          github_token: ${{ secrets.GITHUB_TOKEN }}
          claude_args: |
            --model claude-opus-4-7
            --max-turns 30

      - name: Post result to Asana
        if: always()
        env:
          ASANA_TOKEN: ${{ secrets.ASANA_CI_TOKEN }}
        run: |
          # Post PR link + test summary to Asana task ${{ inputs.task_gid }}
          # See scripts/post-executor-result.sh

Orchestration Layer: Prefect

Why Prefect (not Temporal, not Kestra)

Prefect Temporal Kestra
Learning curve Low (Python decorators) Very high (~1 month) Medium (YAML)
AI/agent workflows ✅ Native Python ⚠️ Lots of boilerplate ❌ YAML gets messy
Self-hosted ✅ Docker, simple ✅ Complex cluster ✅ Docker
Retry / state ✅ Built-in ✅ Indestructible ✅ Kafka-backed
Org-scale RBAC Prefect Cloud (paid) ✅ ✅
Fit for this use case ✅ Best ⚠️ Overkill ⚠️ Wrong paradigm

Temporal is overkill: our tasks complete in <2h, don't need month-long replay guarantees. Kestra is designed for ETL/data, not AI agent orchestration.

Prefect Flow Design

from prefect import flow, task
import subprocess, base64, time

@task(retries=2, retry_delay_seconds=60)
def run_analysis_agent(task_source: str, task_id: str) -> dict:
    """Claude analyzes the task and returns structured executor prompt."""
    # Runs Claude (hermes or claude -p) with task context
    # Returns: {prompt_b64, branch_name, runner_type, confidence}

@task
def dispatch_gha(repo: str, prompt_b64: str, branch: str, runner: str, task_gid: str) -> str:
    """Triggers GHA workflow_dispatch. Returns run_id."""
    result = subprocess.run([
        "gh", "workflow", "run", "executor-worker.yml",
        "--repo", repo,
        "-f", f"executor_prompt={prompt_b64}",
        "-f", f"branch_name={branch}",
        "-f", f"runner_type={runner}",
        "-f", f"task_gid={task_gid}",
    ], capture_output=True, text=True)
    return extract_run_id(result.stdout)

@task(retries=60, retry_delay_seconds=60)
def wait_for_gha(repo: str, run_id: str) -> dict:
    """Polls GHA job until complete. Returns result artifact."""
    status = get_gha_status(repo, run_id)
    if status in ("in_progress", "queued"):
        raise Exception("Still running")  # triggers retry
    return get_result_artifact(repo, run_id)

@task
def post_result(source: str, task_id: str, result: dict):
    """Posts PR link + summary back to Asana/Sentry/MM."""
    ...

@flow(name="executor-task")
def execute_task(source: str, task_id: str, repo: str = "duckduckgo/apple-browsers"):
    analysis = run_analysis_agent(source, task_id)
    run_id = dispatch_gha(repo, analysis["prompt_b64"], analysis["branch"], analysis["runner_type"], task_id)
    result = wait_for_gha(repo, run_id)
    post_result(source, task_id, result)

Prefect Deployment

Self-hosted on Eagle (or VPS) — single Docker container:

# docker-compose.yml
services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    ports: ["4200:4200"]
    environment:
      PREFECT_SERVER_DATABASE_CONNECTION_URL: "postgresql+asyncpg://..."
    volumes:
      - prefect-data:/root/.prefect

  prefect-worker:
    image: prefecthq/prefect:3-latest
    command: prefect worker start --pool "local-process"
    environment:
      PREFECT_API_URL: "http://prefect-server:4200/api"
      ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY}"
      GH_TOKEN: "${GH_TOKEN}"

Prefect UI: http://localhost:4200 — shows all runs, retries, failures, timing.

Sentry AI Analysis → Executor [PLACEHOLDER]

⚠️ Placeholder — implement after Asana direct access is restored. Validate this section against current Sentry bot plan before implementation.

Planned flow:

  1. Sentry receives new crash report matching severity threshold
  2. Webhook triggers Prefect flow: sentry_crash_to_executor
  3. Analysis Agent reads: crash stack trace, affected versions, reproduction frequency, linked Sentry issues
  4. If fix confidence > threshold: dispatch GHA executor with targeted fix prompt
  5. If confidence low: create Asana task with AI-generated analysis, tag "обработать" for human review
  6. Post Sentry comment with AI analysis + PR link (if fix attempted)
Sentry webhook
  → Prefect: sentry_crash_to_executor(issue_id)
      → Analysis Agent: read crash + codebase context
      → confidence ≥ 0.7? → dispatch GHA executor
      → confidence < 0.7? → create Asana task + post Sentry comment
  → Result: PR draft OR Asana task + Sentry analysis comment

Peekaboo on GHA — Constraint

GHA hosted runners run with SIP enabled — tccutil insert for Screen Recording / Accessibility doesn't work.

Solution for visual reproduction:

GHA job (macos-15):
  → build app artifact
  → SSH into ddg-vm on Eagle
  → upload artifact to VM
  → run Peekaboo-based test inside VM
  → retrieve screenshot + result
  → post to PR

ddg-vm is existing infra with Peekaboo pre-installed + TCC permissions granted. This is already how macOS UI Tests CI works today.

Org-Scale Design

For scaling beyond personal use to full org:

Capabilities Registry

Each repo declares its execution capabilities in .github/executor-capabilities.json:

{
  "runner_types": ["macos-15", "ubuntu-latest"],
  "has_ios_build": true,
  "has_ui_tests": true,
  "has_peekaboo_vm": true,
  "max_concurrent_workers": 2
}

Prefect's Analysis Agent reads this to select the correct runner per task.

Shared Infrastructure

Component Where Notes
Prefect server Eagle or VPS Single instance, all repos
ANTHROPIC_CI_KEY GitHub org secrets Shared across repos
ASANA_CI_TOKEN GitHub org secrets Rate-limited bot token
executor-bot (MM) MM Admin → see Asana task Bot account for notifications
ddg-vm pool Eagle 2 VMs available concurrently

Multi-Repo Flows

@flow
def route_task(task_id: str):
    """Routes task to correct repo + executor."""
    repo = classify_repo(task_id)  # Asana tags → repo mapping
    execute_task(source="asana", task_id=task_id, repo=repo)

GHA @claude Trigger (Direct, No Orchestration Needed)

For PR-level tasks, Anthropic's action handles everything natively:

# In any repo: .github/workflows/claude.yml
on:
  issue_comment:
    types: [created]
  pull_request_review_comment:
    types: [created]
jobs:
  claude:
    runs-on: ubuntu-latest
    steps:
      - uses: anthropics/claude-code-action@v1
        with:
          anthropic_api_key: ${{ secrets.ANTHROPIC_CI_KEY }}
          trigger_phrase: "@claude"

Usage: @claude fix the failing unit test in this PR → Claude reads diff, fixes, pushes commit. No Prefect needed — this path is synchronous and triggered by humans.

Phased Implementation

Phase 1: GHA Validation (1-2 days)

  • Create executor-worker.yml in apple-browsers repo
  • Test claude-code-action@v1 with trivial prompt on macOS-15 runner
  • Verify DerivedData cache (target: <10 min cached build)
  • Test ddg-vm SSH from GHA job (Peekaboo path)
  • Add ANTHROPIC_CI_KEY to org secrets

Phase 2: Analysis Agent (2-3 days)

  • Write analysis-agent.md prompt template
  • Test against 3 real Asana bug tasks
  • Tune confidence scoring + runner selection
  • Validate prompt quality → PR quality correlation

Phase 3: Prefect Orchestration (2-3 days)

  • Deploy Prefect server (Docker on Eagle or VPS)
  • Implement execute_task flow
  • Wire Asana "обработать" tag → Prefect trigger
  • Test end-to-end: Asana task → PR

Phase 4: MM Integration (1 day)

  • Create executor-bot in MM (blocked by IT Ops — see Asana task)
  • Add MM notification step to Prefect flow
  • Create #executor-queue channel

Phase 5: Sentry Integration [PLACEHOLDER]

  • Validate against current Sentry bot plan (requires Asana access restoration)
  • Implement sentry_crash_to_executor flow
  • Set confidence threshold for auto-fix vs. triage

Phase 6: Org Rollout

  • Publish executor-capabilities.json spec
  • Template executor-worker.yml as reusable workflow
  • Onboard second repo (e.g. privacy-configuration)
  • Add route_task multi-repo flow to Prefect

Research Findings (2026-05-14)

GHA macOS Runners

  • macos-15 = current macos-latest (since Sep 2025)
  • Xcode: max 3 simulator runtimes per image
  • Simulators: headless, no display server needed → XCUITests work
  • Build cache: DerivedData cacheable via actions/cache

Peekaboo on GHA

  • Requires Screen Recording + Accessibility TCC permissions
  • SIP enabled on hosted runners → tccutil insert blocked
  • Viable path: ddg-vm SSH from GHA job (existing infra)

claude-code-action@v1 (GA)

  • Official Anthropic action, replaces manual claude -p
  • workflow_dispatch trigger works for external orchestration
  • --model claude-opus-4-7 for complex fixes, sonnet for quick ones
  • GH token scoped to repo → no additional secrets needed for PR creation

Prefect 3.x

  • Python decorators, low learning curve
  • Self-hosted: single Docker container + Postgres
  • Prefect Cloud free tier: 3 workspaces, unlimited flows (but limited runs/month)
  • Retry pattern for GHA polling: @task(retries=60, retry_delay_seconds=60)

Notes

  • Eagle's current cron executor tick continues to run in parallel during transition
  • GHA macos-15 runner: Xcode NOT pre-cached → first build ~30 min, cached ~8 min
  • Self-hosted runner on Eagle: avoids cache miss but ties up main Mac
  • DuckDuckGo has existing GHA macOS setup — reuse existing Xcode bootstrap steps
  • ANTHROPIC_CI_KEY = separate key from personal key, with usage limits
Reference in New Issue View Git Blame Copy Permalink