From e98d8ba61f4cdd628afcf0279d523cbe25fc08d3 Mon Sep 17 00:00:00 2001 From: Alexey Martemyanov Date: Fri, 16 Oct 2026 17:24:28 +0600 Subject: [PATCH] [2026-10-16] eagle: personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/telegram-approvals.md --- .../docs/operations.md | 229 ++++++++++++++++++ .../docs/telegram-approvals.md | 160 ++++++++++++ 2 files changed, 389 insertions(+) create mode 100644 personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md create mode 100644 personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/telegram-approvals.md diff --git a/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md b/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md new file mode 100644 index 00000000..bc4cfc1c --- /dev/null +++ b/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md @@ -0,0 +1,229 @@ +# Архив исходника Broker + +Историческая копия из `ae92c15` до очистки репозитория 2026-10-16; текущая установка — [[personal/projects/balda/execution-broker]]. + +--- + +# Operations and recovery + +Use `executionctl --token-file … --operation tools` for authenticated MCP reachability and `execution_status` for a specific run. `/healthz` is liveness only. JSON logs include run IDs and retry causes, without configured tokens. Virfield maintenance/capacity may keep a run provisioning until its deadline; inspect both services rather than creating another idempotency key. + +For the local Compose stack, `deploy/compose.dynamic-images.yaml` pins selected +image tags. Compare those tags with the running containers before claiming a +source change is deployed. A Git push does not rebuild or restart the stack. + +For Broker-managed local Git projects, macOS bootstrap automatically scopes the +guest configuration to the run's exact registered repository and enables its +private SSH Git tunnel. It retains the reviewed package digest and workflow pin; +new project names do not require a worker archive rebuild or manual allowlist. +The installed marker covers both the immutable archive and the effective guest +configuration, so a retry repairs an old guest configuration before starting. +Broker checks package workflow/profile compatibility before VM acquisition. +Permanent worker start refusals, including `forbidden`, `workflow_integrity` and +`bootstrap_incompatible`, end the run as failed and notify the original Balda +conversation with diagnostics. Temporary infrastructure failures retain retry +backoff; Telegram approval routing does not suppress terminal errors. + +Project tools use [operator-owned tooling profiles](tooling.md). An operator +`default_tooling` entry can preserve a worker's existing standard Go environment; +an explicit task selection pins another approved recipe. SDKs are size/SHA-checked +and installed only in the disposable run directory, with their observed versions +retained in `tooling.json`. Golden images and the operator host are not modified. +Make/C/C++, Python, Node and JVM use the same package/archive/system backends. + +Enable the independent [repository cache](repository-cache.md) only for approved +sources. [Environment changes](environment-changes.md) require an operator-authored +safe checkpoint and continuation in the pinned workflow pack; arbitrary graphs +are not resumed by replay. Deploy the matching execution/v3 gateway, Runner and +bootstrap before enabling either policy. An environment-change terminal outcome +must complete old-worker/credential cleanup before its separately approved +replacement can allocate anything. Preserve its original deadline and artifacts. + +New workers persist whether provider credentials could have been used, and +record a stopped fence only after the Callee process group settles. An unused +preparation lease preserves the issuer cache. Missing admission/process metadata, +a missing used cache or an unknown supervisor outcome leaves the exclusive owner +locked for recovery; an empty phase is not evidence that credentials were unused. +Inspect `credential_state_lost`, `process_group_unsettled` and `cleanup_error` +instead of starting another attempt with the same account. + +If a run ends with `callee_failed`, first read `execution_diagnostics` using its +bound chat and topic IDs. On updated Broker and worker builds, `failure_detail` +includes the bounded final adapter failure explanation from a checksum-verified +`callee.log`, even when no `quality.json` was produced. Earlier transport failure +classes still distinguish a missing result, process exit or signal, malformed +event, or output read failure. Quality diagnostics retain review and verification +findings separately. +Check the last role event and the `callee.log` artifact to locate the boundary; +an empty log or generic `callee_failed` does not prove an agent started or +crashed. Balda must report the cause as unrecorded when diagnostics has neither +a quality finding nor `failure_detail`; it must not guess from an empty artifact. +For Broker-terminal failures such as `lease_failed`, the terminal code replaces +any earlier retry code in `last_error`. Historical runs that ended before this +fix can still show a stale retry code; inspect Virfield lease events and the +prepare job for the actual resource failure. Older deployed builds may not +expose `failure_detail`; update Broker +and the worker package before relying on it. The Callee adapter accepts an +approval question up to 256 KiB. Runner and the journal retain a complete Human +event up to 512 KiB after JSON serialization; ordinary event summaries retain +their 32 KiB limit. Runner event-processing errors are redacted and saved in +`callee.log` before artifact collection, so their concrete cause is available to +`execution_diagnostics`. A +confirmed old 16 KiB +prompt cap requires an updated VM bootstrap package before retrying with a new +Story/run key. A Story created under placeholder `0/0` cannot sync with the real +Balda Session ID; create a new Story and Task bound to the current Session ID. + +## VM readiness failures + +`guest_keychain_failed` is a Virfield preparation failure before the worker is +ready. Its public error covers native GUI/SSH key storage and unlocked, +unobstructed console checks; it does not identify a password, OAuth or provider +failure. A failed prerequisite can leave no Runner/Callee log or artifacts. +Correlate the Broker run with the exact Virfield lease, prepare job and events. +Preserve bounded private provisioning diagnostics before cleanup when available, +within the existing deadline; never keep a failed lease indefinitely for logs. +After release its private directory may already be gone. Retain the public +failure and confirmed release, and record the unavailable diagnostic scope. + +The [support matrix](support.md#runtime-matrix) records the production macos27 +reacceptance passed on 2026-10-16 and preserves the three earlier failed probes. +The dated [Virfield record](https://github.com/mallexxx/virfield/blob/a1460d3372bdfbb5cb19d633d41dd5d3fb3164e6/docs/VERIFICATION.md#broker-macos27-reacceptance) +defines the tested scope; exact run/job/lease references remain in the private +operator incident register. An idle pool, a catalog entry or a different +OS/profile's success does not qualify another deployment. + +On macOS, `kill(-pgid, SIGKILL)` can return `EPERM` for a group containing only +zombies. Runner settles that case only after a separate bounded process +observation finds no executable group members. An observation error or a live +member retains the credential fence; do not clear ownership from the signal +error alone. + +## Recovery guarantees + +Broker stores admission, stable acquisition inputs, every transition and events in SQLite WAL with FULL synchronous writes. It holds a process lock on its data directory. Run one Broker process per database on a local reliable filesystem; SQLite on NFS or multiple independent replicas is unsupported. Newer database schema versions fail closed. + +Restart Broker freely: it replays uncertain acquisition/start with the same identity, redelivers unacknowledged Human answers and retrieves missed events. A running Runner stays alive inside its worker when Broker restarts. If the Runner process itself dies, it records a diagnostic failed outcome rather than rerunning an unknown workflow. + +For cancellation call `execution_cancel`, retain the run ID, and wait for terminal **and** `cleanup=done`. VM cleanup belongs to Virfield, including Broker downtime. Docker Runner enforces its deadline; gateway independently reaps owned workers one hour after their lease deadline. This grace period permits artifact recovery after an outage. Resource slots remain occupied until release is confirmed. Docker and host-runner retain durable revocation records, so a delayed start cannot recreate a released run. Virfield admission refusals are distinguished from unknown acquisition outcomes; canceling a proven refusal does not create a VM just to delete it. Never manually remove resources belonging to another run to make capacity. + +Artifacts resume from a private on-disk prefix after transfer interruption and Broker restart; the complete file is SHA-verified and fsynced before terminal publication. References expire after seven days. An hourly bounded retention pass removes expired payload directories only after confirmed worker cleanup and expiry of every reference. It preserves event, checksum, Story and idempotency history. Historical SQLite/Beads journals remain durable; monitor and size these separately. Partial unpublished transfers are reclaimed seven days after the original execution deadline. Loss of disk space fails transactions and retries without reporting success. Execution stops at its original deadline. Broker and Virfield reserve at most five additional minutes to collect already-written results; provider execution is not extended. Worker-side collection has its own one-minute bound. A lost worker before artifact copy can leave the run awaiting recovery until that bound; missing artifacts are not silently fabricated. + +Bootstrap archives in the deployment `assets` directory and local `.build` +outputs have no automatic retention pass. After every run pinned to an old +package has reached terminal state and `cleanup=done`, compare all configured +worker `bootstrap_file` paths with the archive directory before removing old +packages. Check `df` after cleanup: APFS clones and local Time Machine snapshots +can retain blocks even when `du` for the directory shrinks. Virfield reserves +the VM's full virtual disk size plus its host safety reserve at admission; a +run can be rejected while the image's currently allocated size is smaller. + +## Backup and restore + +Use `execution-broker --config … --backup /absolute/new-backup.db` for a consistent live SQLite snapshot. The destination must not exist. Copy the Broker artifact directory alongside the snapshot; pause new admission for a fully coordinated database/artifact backup. Transport keys are included in the private database: encrypt backups and restrict access to the service administrator. + +For Prism/Beads backup, stop the Prism sidecar and take consistent snapshots of **both** `prism-state` and `beads-state` volumes, then start it again. The projection database retains facts/cursors and the Beads directory retains Dolt history; backing up only one is not a complete recovery point. + +Restore into an empty data directory with the same original worker profiles, bearer secrets and approval public keys. Stop the existing instance first. Restore the database/artifacts with restricted permissions and restart one service. Verify pending run statuses and resource cleanup before accepting new work. Do not restore an old snapshot while the original instance is still accepting requests; lost post-snapshot idempotency history can duplicate external work. + +## Balda delivery + +Prism projection and chat delivery have independent cursors. An adapter ack means Beads facts were applied, not that a chat message was delivered. Optional Broker `notifications` routes call Balda's durable inbound webhook with an idempotency header; the destination must deduplicate that header. By default, a 2xx advances the notification cursor. With `require_delivery_ack=true`, a Balda `202` leaves the event pending and Broker retries the same key; only `200` with a durable provider receipt advances the cursor. The payload contains only run/Story/session/event references; Balda reads authoritative state through MCP before delivering a response. A route may opt into `coalesce_cleanup=true`: Broker persists whether cleanup was already `done` before the first terminal webhook attempt. Only that case permits advancing a later `cleanup_completed` event without another post. Cleanup that finishes during asynchronous delivery is notified separately, including after a Broker restart; a later receipt cannot retroactively change what the terminal message said. The default remains separate notifications. + +The packaged Balda skill persists each attempt and arms a native original-session +one-shot wait before execution admission. It uses `execution_lookup` after a lost +Start/retry response and closes abandoned admissions atomically before declaring +that no execution can arrive later. Each wake arms a fresh generation before +reading Broker state; a separate TTL recovery wait covers a failed intermediate +model turn. Consult [Balda integration](../integrations/balda/README.md) for the +upgrade contract. A wait acknowledgement is not a chat-provider delivery receipt. +Monitor failed observer turns and retain run IDs for manual recovery. + +Current generic Balda webhooks execute in isolated private jobs. Retired session +envelope fields are invalid; a wildcard route cannot continue arbitrary sessions. +Disable obsolete notification routes while preserving their cursors and audit +history. Do not mark retained notifications delivered as part of a migration. +Existing Balda sessions pin capability snapshots; a new conversation adopts the +upgraded skill without modifying an existing conversation's history. + +## Story quality evidence + +Implementation, verification and review run after the deterministic owner approval +gate. Their authored context identifies that approval as a workflow-journal fact, +not a repository file. Worker checkouts intentionally use detached HEAD at the +pinned starting revision; local commits are exported in `commits.bundle` and do +not require a named branch or an upstream push. + +Every verifier check is a repeatable shell assertion with `exit_code=0` and +`expected_exit_code=0` (the default). A negative CLI behavior check wraps the +application invocation in an assertion of its exit code and output; the assertion +itself must succeed. The deterministic quality gate repeats each assertion in +the same disposable checkout and retains its actual exit and bounded output. +It never accepts a failed suite because the model declared that failure expected. +A mismatch, unexecuted check, unresolved +acceptance finding or malformed report fails the quality gate and prevents code +publication. Non-blocking observations belong in optional string `notes`. + +## One worker per run and bounded repairs + +A worker belongs to a run, and a terminal outcome releases it after artifact +collection/checkpoint. An unfinished Prism Story does not by itself retain a VM. +Coding workflows for Codex, Claude and OpenCode keep implementation, verification +and review in the same checkout for up to three repair iterations. Failed test +or review reports feed the next implementation session without a new run, VM or +plan-approval replay. The final quality gate still rejects an exhausted cycle. +An intentional new attempt with the same Story, originating chat/topic, task, +repository revision, workflow ID, worker, image and agent reuses the exact +approved plan text from its durable Human journal. It starts at the authored +implementation/repair/quality subtree rather than regenerating a plan. A new +idempotency key and repaired workflow pack do not invalidate that decision; +changed execution scope or later owner rejection/amendment requires approval. +Approval snapshots travel only in private worker manifests and are omitted from +public status. ACP permission and provider sign-in requests remain independent. +Infrastructure/process failures, explicit cancellation and the original deadline +remain terminal; they are not indefinite VM retention or blind workflow replay. + +Runner and its separate RPC processes open the same SQLite journal. Write +transactions reserve the writer before reading the update snapshot +(`BEGIN IMMEDIATE`), avoiding a deferred-transaction SQLITE_BUSY_SNAPSHOT (517). +The busy timeout is initialized per connection before normal database setup. + +`execution_diagnostics` reads the checksum-verified Runner/Callee log for every +unsuccessful terminal state, including expired and canceled runs. For example, +an approval wait that reaches the original run deadline reports +`context deadline exceeded` rather than only `callee_event_processing_failed`. +Logs produced before Runner recorded the underlying error cannot reconstruct +that missing cause retrospectively. + +If Claude clears every native OAuth token after an authentication rejection, +the stopped-worker checkpoint retains its account/organization binding and +marks the issuer cache as requiring sign-in. The issuer preserves the previous +encrypted bytes, permits cleanup of that exact worker, and does not reissue the +rejected cache. The next attempt uses the existing browser authentication Human +before allocating a VM; completing sign-in resumes it with the retained plan. +A missing, locked or partially corrupt cache still fails closed. Cleanup retries +use `cleanup_error` and never overwrite the recorded execution failure. + +## Retained input dependencies + +Inputs/tooling require a consistent Broker, gateway, Runner and bootstrap set. +Drain v2 runs before rolling back binaries; older builds can discard new fields +when reading state. Restore a prior database only as part of an explicitly chosen +consistent recovery point, accounting for all admissions after that point. + +Use `execution_start.inputs` only with the exact source run, artifact name and +SHA256, in the original chat/topic. The source must have completed cleanup and +still be retained when admitting the dependency. Retention automatically holds +its bytes until dependent cleanup or the original dependent deadline plus five +minutes; restart does not extend this limit. A canceled admission takes no hold. + +`artifact_input_failed` includes the concrete failed transfer prerequisite in the +terminal detail. Changed ETag/bytes, inaccessible sources, invalid ranges and +redirects stop preparation before Callee starts. A lost Runner process may resume +only its explicit input preparation checkpoint. Do not relabel a lost active +Callee process as an input checkpoint to force a replay. + +The internal `/worker-inputs/{run_id}/{name}` route accepts only the scoped, +expiring worker token and supports byte ranges. It is not a public artifact link. +Keep the normal Broker token out of workers. Input preparation does not install +tools, run an archive's contents, or expand a managed-host approval. diff --git a/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/telegram-approvals.md b/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/telegram-approvals.md new file mode 100644 index 00000000..a5dfbd2e --- /dev/null +++ b/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/telegram-approvals.md @@ -0,0 +1,160 @@ +# Архив исходника Broker + +Историческая копия из `ae92c15` до очистки репозитория 2026-10-16; текущая установка — [[personal/projects/balda/execution-broker]]. + +--- + +# Telegram approval actions + +A retry of the same Story and exact execution scope retains the actual plan +previously approved by the owner. Broker loads that decision from its journal +and sends a private, checksum-bound snapshot to Runner/Callee. Callee continues +implementation and the quality gate without regenerating planning or asking for +the same approval again. Changes to the task, repository revision, executor or +session require a new approval. Later owner feedback is never replaced with an +older approval. ACP permission choices and command authorization remain separate. + +Broker can deliver pending workflow and worker ACP permission requests to a +dedicated Telegram bot, even when the run started in Mattermost. Balda's +existing notification route remains responsible for the original conversation +and terminal result. Telegram never creates a new Story, run or Balda session. +The original Balda conversation is the default approval channel. When Telegram +is enabled for a source session, approval requests are delivered only to Telegram; +Balda does not repeat the question or options. Result and cleanup notifications +continue in the original conversation. The existing command authorization API +remains available for operator recovery. Authored command allow/deny/ask +policies, bearer authentication and operator grants retain their existing roles. + +## Configure + +Create a dedicated bot with BotFather. Save its token in the deployment's private +`telegram-approval.token` file, mode 0600; never put it in Git or a plugin. +Set the owner's numeric Telegram user ID from an independently established +identity. The bot does not enroll the first user who contacts it. The owner +must open the bot's private chat and press Start before it can deliver requests. + +Add this operator-owned block to `broker.json`: + +```json +{ + "telegram_approvals": { + "enabled": true, + "token_file": "/run/secrets/telegram_approval_token", + "owner_user_id": 123456789, + "chat_ids": ["mm-c-EXACT_SOURCE_SESSION"], + "timeout_seconds": 1800, + "min_run_ttl_seconds": 3600 + } +} +``` + +Omit this block or set `"enabled": false` to use the current approval method +without starting a Telegram client, reading its token or requiring a bot secret. +Only `"enabled": true` selects the Telegram approval route. The Bot API transport +does not replace command authorization or change the original approval API. + +This example allows 30 minutes for a Telegram decision and requires at least one +hour of total run TTL, leaving time for planning and execution. For selected +source sessions, `min_run_ttl_seconds` rejects a new shorter `execution_start` +before admission or worker allocation; retry the same unaccepted request with +an adequate `ttl_seconds`. Zero or omission preserves the caller-selected TTL. +Already admitted requests retain their original deadline and idempotent replay. +The execution deadline remains a hard bound, including queueing and planning; +it is never extended when a question appears or the owner refreshes controls. + +`chat_ids` identifies original Balda sessions. An explicit `"*"` selects all +sessions in this single-operator Broker. It does not enable multi-user approval. +Include `deploy/compose.telegram-approvals.yaml`, rebuild the Broker image, +select that reviewed image in an operator override, and recreate only Broker +after checking current runs and retaining a rollback image/configuration. +The default deployment image pins remain unchanged. The bot uses outgoing long +polling; no new public listener is needed. A configured Telegram webhook prevents +polling and is never removed +automatically. Run only one consumer for this bot token. + +## Actions and ownership + +- Workflow approval offers Approve and Cancel. Reply to the exact prompt to amend + the requirements where the selected workflow supports amendments. +- Updated Callee adapters send structured ACP option IDs and labels. The buttons + offer only the provider's actual choices, such as Allow Once and Deny. Authored + `allow` and `deny` roles retain their existing policy; only `ask` requests pause. + Update worker packages as well as Broker to obtain structured ACP actions. +- Every prompt also offers Cancel run. Unknown or older generic questions accept + a reply to their exact Telegram message instead of inventing an approval value. + Known installed Story packages retain their existing approved/cancel contract. +- Only the configured owner's user ID in that owner's private chat can decide. + Callback data contains an unpredictable, durable request nonce and an option + index. It never contains credentials, paths, shell commands or approval answers. +- Broker submits the selected answer against the original run, Human request, + chat and topic. The accepted-response journal records the Telegram user ID. + Concurrent operator recovery and Telegram responses accept one decision; a + stale cancellation button cannot cancel an already answered request. + +Each request sends one readable message with its action buttons. Workflow plans +use the task's language and omit internal Story/run/request IDs. A long question +is explicitly marked as a preview and offers **Full details** (Russian: **Полный +план**). Only clicking that button sends the complete UTF-8 `approval.txt` file; +it does not approve, cancel or resume the worker. Repeated clicks reuse the +recorded document delivery. Read the complete details before approving a preview. + +Reply to the message with buttons to change a Story plan; a standalone message +or a reply to its attachment is not an approval response. Up to three requirement +amendments regenerate planning in the same run and ask for approval again. +ACP permission prompts accept only their exact offered option IDs; text feedback +is not a requirement amendment for those prompts. + +## Recovery and expiration + +Prompt bindings, delivery IDs and deadlines live in the private Broker run state. +The Bot API update cursor is atomically saved in the Broker data directory after +processing an action. A prompt is sent without buttons, its message ID is saved, +and only then are controls attached. Restart completes an interrupted control +publication on that same message. An ambiguous send may leave a duplicate text +message without buttons; it cannot leave an untracked actionable prompt. +Duplicate callbacks do not resume a worker twice. Resolved controls are removed; +clicking an older actionable copy also removes that copy's controls before the +update is acknowledged. Already expired callback acknowledgements do not block +later updates. + +The Telegram timeout is bounded by the original execution deadline. Expiry removes +the approval controls and rejects their late actions. The same message explains +expiration and offers **Refresh request**. Only the configured owner can refresh +a still-pending Human; delivery creates a fresh nonce and asks for a new decision +in the same run and worker. The renewal request is saved before clearing the old +controls; after a restart, delivery finishes renewal without another callback. +It does not accept an approval, extend the execution +deadline, or send the request into Mattermost. Stale callbacks, changed Human +requests and expired executions cannot refresh it. The existing command API +also remains available until the normal run deadline. Telegram +failures or expiry do not send a duplicate question into the original conversation. +Disabling Telegram restores conversation delivery for a still-pending request. +An explicit Cancel run button still requests cancellation. +Replies to stale prompts, other users, group messages, +unknown actions and changed request bindings cannot resume execution. + +Provider sign-in challenges are excluded from this route. Managed-host signed +capabilities and folder-access operator grants retain their independent +authorization paths. This feature covers Broker-managed runs; Balda's own host +ACP permission service is a separate integration. The context-review subagent +and policy-based autoapproval are separate planned work. + +## Verification + +Run `make check` and `scripts/check-callee.sh`. Tests cover allow/deny actions, +exclusive approval delivery with Telegram enabled or disabled, bot failures, +active status/event polling, switching a pending request back to the conversation, +and continued terminal/result delivery, as well as +the exact worker response, competing channels, unauthorized and stale clicks, +timeout, restart, ambiguous sends, interrupted control publication and renewal, +stale-copy cleanup failures, acknowledgement failure, complete documents, +requirement amendments, secret-safe transport errors and private API projection. +On 2026-10-12, candidate `05358d0` passed an isolated Docker acceptance run +with the real Approval bot. The configured owner clicked Approve; Broker recorded +one attributed response, the deterministic Callee fixture succeeded, all five +artifact hashes matched, and worker container/volume cleanup was independently +verified. The test stack had no Balda notification route, so this does not prove +Mattermost delivery or a real-provider ACP permission request. The production +rollout and source-session scope require their own deployment verification. + +Bot transport contracts: [Telegram Bot API](https://core.telegram.org/bots/api).