obsidian-vault/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md

# Архив исходника Broker

Историческая копия из `ae92c15` до очистки репозитория 2026-10-16; текущая установка — [[personal/projects/balda/execution-broker]].

---

# Operations and recovery

Use `executionctl --token-file … --operation tools` for authenticated MCP reachability and `execution_status` for a specific run. `/healthz` is liveness only. JSON logs include run IDs and retry causes, without configured tokens. Virfield maintenance/capacity may keep a run provisioning until its deadline; inspect both services rather than creating another idempotency key.

For the local Compose stack, `deploy/compose.dynamic-images.yaml` pins selected
image tags. Compare those tags with the running containers before claiming a
source change is deployed. A Git push does not rebuild or restart the stack.

For Broker-managed local Git projects, macOS bootstrap automatically scopes the
guest configuration to the run's exact registered repository and enables its
private SSH Git tunnel. It retains the reviewed package digest and workflow pin;
new project names do not require a worker archive rebuild or manual allowlist.
The installed marker covers both the immutable archive and the effective guest
configuration, so a retry repairs an old guest configuration before starting.
Broker checks package workflow/profile compatibility before VM acquisition.
Permanent worker start refusals, including `forbidden`, `workflow_integrity` and
`bootstrap_incompatible`, end the run as failed and notify the original Balda
conversation with diagnostics. Temporary infrastructure failures retain retry
backoff; Telegram approval routing does not suppress terminal errors.

Project tools use [operator-owned tooling profiles](tooling.md). An operator
`default_tooling` entry can preserve a worker's existing standard Go environment;
an explicit task selection pins another approved recipe. SDKs are size/SHA-checked
and installed only in the disposable run directory, with their observed versions
retained in `tooling.json`. Golden images and the operator host are not modified.
Make/C/C++, Python, Node and JVM use the same package/archive/system backends.

Enable the independent [repository cache](repository-cache.md) only for approved
sources. [Environment changes](environment-changes.md) require an operator-authored
safe checkpoint and continuation in the pinned workflow pack; arbitrary graphs
are not resumed by replay. Deploy the matching execution/v3 gateway, Runner and
bootstrap before enabling either policy. An environment-change terminal outcome
must complete old-worker/credential cleanup before its separately approved
replacement can allocate anything. Preserve its original deadline and artifacts.

New workers persist whether provider credentials could have been used, and
record a stopped fence only after the Callee process group settles. An unused
preparation lease preserves the issuer cache. Missing admission/process metadata,
a missing used cache or an unknown supervisor outcome leaves the exclusive owner
locked for recovery; an empty phase is not evidence that credentials were unused.
Inspect `credential_state_lost`, `process_group_unsettled` and `cleanup_error`
instead of starting another attempt with the same account.

If a run ends with `callee_failed`, first read `execution_diagnostics` using its
bound chat and topic IDs. On updated Broker and worker builds, `failure_detail`
includes the bounded final adapter failure explanation from a checksum-verified
`callee.log`, even when no `quality.json` was produced. Earlier transport failure
classes still distinguish a missing result, process exit or signal, malformed
event, or output read failure. Quality diagnostics retain review and verification
findings separately.
Check the last role event and the `callee.log` artifact to locate the boundary;
an empty log or generic `callee_failed` does not prove an agent started or
crashed. Balda must report the cause as unrecorded when diagnostics has neither
a quality finding nor `failure_detail`; it must not guess from an empty artifact.
For Broker-terminal failures such as `lease_failed`, the terminal code replaces
any earlier retry code in `last_error`. Historical runs that ended before this
fix can still show a stale retry code; inspect Virfield lease events and the
prepare job for the actual resource failure. Older deployed builds may not
expose `failure_detail`; update Broker
and the worker package before relying on it. The Callee adapter accepts an
approval question up to 256 KiB. Runner and the journal retain a complete Human
event up to 512 KiB after JSON serialization; ordinary event summaries retain
their 32 KiB limit. Runner event-processing errors are redacted and saved in
`callee.log` before artifact collection, so their concrete cause is available to
`execution_diagnostics`. A
confirmed old 16 KiB
prompt cap requires an updated VM bootstrap package before retrying with a new
Story/run key. A Story created under placeholder `0/0` cannot sync with the real
Balda Session ID; create a new Story and Task bound to the current Session ID.

## VM readiness failures

`guest_keychain_failed` is a Virfield preparation failure before the worker is
ready. Its public error covers native GUI/SSH key storage and unlocked,
unobstructed console checks; it does not identify a password, OAuth or provider
failure. A failed prerequisite can leave no Runner/Callee log or artifacts.
Correlate the Broker run with the exact Virfield lease, prepare job and events.
Preserve bounded private provisioning diagnostics before cleanup when available,
within the existing deadline; never keep a failed lease indefinitely for logs.
After release its private directory may already be gone. Retain the public
failure and confirmed release, and record the unavailable diagnostic scope.

The [support matrix](support.md#runtime-matrix) records the production macos27
reacceptance passed on 2026-10-16 and preserves the three earlier failed probes.
The dated [Virfield record](https://github.com/mallexxx/virfield/blob/a1460d3372bdfbb5cb19d633d41dd5d3fb3164e6/docs/VERIFICATION.md#broker-macos27-reacceptance)
defines the tested scope; exact run/job/lease references remain in the private
operator incident register. An idle pool, a catalog entry or a different
OS/profile's success does not qualify another deployment.

On macOS, `kill(-pgid, SIGKILL)` can return `EPERM` for a group containing only
zombies. Runner settles that case only after a separate bounded process
observation finds no executable group members. An observation error or a live
member retains the credential fence; do not clear ownership from the signal
error alone.

## Recovery guarantees

Broker stores admission, stable acquisition inputs, every transition and events in SQLite WAL with FULL synchronous writes. It holds a process lock on its data directory. Run one Broker process per database on a local reliable filesystem; SQLite on NFS or multiple independent replicas is unsupported. Newer database schema versions fail closed.

Restart Broker freely: it replays uncertain acquisition/start with the same identity, redelivers unacknowledged Human answers and retrieves missed events. A running Runner stays alive inside its worker when Broker restarts. If the Runner process itself dies, it records a diagnostic failed outcome rather than rerunning an unknown workflow.

For cancellation call `execution_cancel`, retain the run ID, and wait for terminal **and** `cleanup=done`. VM cleanup belongs to Virfield, including Broker downtime. Docker Runner enforces its deadline; gateway independently reaps owned workers one hour after their lease deadline. This grace period permits artifact recovery after an outage. Resource slots remain occupied until release is confirmed. Docker and host-runner retain durable revocation records, so a delayed start cannot recreate a released run. Virfield admission refusals are distinguished from unknown acquisition outcomes; canceling a proven refusal does not create a VM just to delete it. Never manually remove resources belonging to another run to make capacity.

Artifacts resume from a private on-disk prefix after transfer interruption and Broker restart; the complete file is SHA-verified and fsynced before terminal publication. References expire after seven days. An hourly bounded retention pass removes expired payload directories only after confirmed worker cleanup and expiry of every reference. It preserves event, checksum, Story and idempotency history. Historical SQLite/Beads journals remain durable; monitor and size these separately. Partial unpublished transfers are reclaimed seven days after the original execution deadline. Loss of disk space fails transactions and retries without reporting success. Execution stops at its original deadline. Broker and Virfield reserve at most five additional minutes to collect already-written results; provider execution is not extended. Worker-side collection has its own one-minute bound. A lost worker before artifact copy can leave the run awaiting recovery until that bound; missing artifacts are not silently fabricated.

Bootstrap archives in the deployment `assets` directory and local `.build`
outputs have no automatic retention pass. After every run pinned to an old
package has reached terminal state and `cleanup=done`, compare all configured
worker `bootstrap_file` paths with the archive directory before removing old
packages. Check `df` after cleanup: APFS clones and local Time Machine snapshots
can retain blocks even when `du` for the directory shrinks. Virfield reserves
the VM's full virtual disk size plus its host safety reserve at admission; a
run can be rejected while the image's currently allocated size is smaller.

## Backup and restore

Use `execution-broker --config … --backup /absolute/new-backup.db` for a consistent live SQLite snapshot. The destination must not exist. Copy the Broker artifact directory alongside the snapshot; pause new admission for a fully coordinated database/artifact backup. Transport keys are included in the private database: encrypt backups and restrict access to the service administrator.

For Prism/Beads backup, stop the Prism sidecar and take consistent snapshots of **both** `prism-state` and `beads-state` volumes, then start it again. The projection database retains facts/cursors and the Beads directory retains Dolt history; backing up only one is not a complete recovery point.

Restore into an empty data directory with the same original worker profiles, bearer secrets and approval public keys. Stop the existing instance first. Restore the database/artifacts with restricted permissions and restart one service. Verify pending run statuses and resource cleanup before accepting new work. Do not restore an old snapshot while the original instance is still accepting requests; lost post-snapshot idempotency history can duplicate external work.

## Balda delivery

Prism projection and chat delivery have independent cursors. An adapter ack means Beads facts were applied, not that a chat message was delivered. Optional Broker `notifications` routes call Balda's durable inbound webhook with an idempotency header; the destination must deduplicate that header. By default, a 2xx advances the notification cursor. With `require_delivery_ack=true`, a Balda `202` leaves the event pending and Broker retries the same key; only `200` with a durable provider receipt advances the cursor. The payload contains only run/Story/session/event references; Balda reads authoritative state through MCP before delivering a response. A route may opt into `coalesce_cleanup=true`: Broker persists whether cleanup was already `done` before the first terminal webhook attempt. Only that case permits advancing a later `cleanup_completed` event without another post. Cleanup that finishes during asynchronous delivery is notified separately, including after a Broker restart; a later receipt cannot retroactively change what the terminal message said. The default remains separate notifications.

The packaged Balda skill persists each attempt and arms a native original-session
one-shot wait before execution admission. It uses `execution_lookup` after a lost
Start/retry response and closes abandoned admissions atomically before declaring
that no execution can arrive later. Each wake arms a fresh generation before
reading Broker state; a separate TTL recovery wait covers a failed intermediate
model turn. Consult [Balda integration](../integrations/balda/README.md) for the
upgrade contract. A wait acknowledgement is not a chat-provider delivery receipt.
Monitor failed observer turns and retain run IDs for manual recovery.

Current generic Balda webhooks execute in isolated private jobs. Retired session
envelope fields are invalid; a wildcard route cannot continue arbitrary sessions.
Disable obsolete notification routes while preserving their cursors and audit
history. Do not mark retained notifications delivered as part of a migration.
Existing Balda sessions pin capability snapshots; a new conversation adopts the
upgraded skill without modifying an existing conversation's history.

## Story quality evidence

Implementation, verification and review run after the deterministic owner approval
gate. Their authored context identifies that approval as a workflow-journal fact,
not a repository file. Worker checkouts intentionally use detached HEAD at the
pinned starting revision; local commits are exported in `commits.bundle` and do
not require a named branch or an upstream push.

Every verifier check is a repeatable shell assertion with `exit_code=0` and
`expected_exit_code=0` (the default). A negative CLI behavior check wraps the
application invocation in an assertion of its exit code and output; the assertion
itself must succeed. The deterministic quality gate repeats each assertion in
the same disposable checkout and retains its actual exit and bounded output.
It never accepts a failed suite because the model declared that failure expected.
A mismatch, unexecuted check, unresolved
acceptance finding or malformed report fails the quality gate and prevents code
publication. Non-blocking observations belong in optional string `notes`.

## One worker per run and bounded repairs

A worker belongs to a run, and a terminal outcome releases it after artifact
collection/checkpoint. An unfinished Prism Story does not by itself retain a VM.
Coding workflows for Codex, Claude and OpenCode keep implementation, verification
and review in the same checkout for up to three repair iterations. Failed test
or review reports feed the next implementation session without a new run, VM or
plan-approval replay. The final quality gate still rejects an exhausted cycle.
An intentional new attempt with the same Story, originating chat/topic, task,
repository revision, workflow ID, worker, image and agent reuses the exact
approved plan text from its durable Human journal. It starts at the authored
implementation/repair/quality subtree rather than regenerating a plan. A new
idempotency key and repaired workflow pack do not invalidate that decision;
changed execution scope or later owner rejection/amendment requires approval.
Approval snapshots travel only in private worker manifests and are omitted from
public status. ACP permission and provider sign-in requests remain independent.
Infrastructure/process failures, explicit cancellation and the original deadline
remain terminal; they are not indefinite VM retention or blind workflow replay.

Runner and its separate RPC processes open the same SQLite journal. Write
transactions reserve the writer before reading the update snapshot
(`BEGIN IMMEDIATE`), avoiding a deferred-transaction SQLITE_BUSY_SNAPSHOT (517).
The busy timeout is initialized per connection before normal database setup.

`execution_diagnostics` reads the checksum-verified Runner/Callee log for every
unsuccessful terminal state, including expired and canceled runs. For example,
an approval wait that reaches the original run deadline reports
`context deadline exceeded` rather than only `callee_event_processing_failed`.
Logs produced before Runner recorded the underlying error cannot reconstruct
that missing cause retrospectively.

If Claude clears every native OAuth token after an authentication rejection,
the stopped-worker checkpoint retains its account/organization binding and
marks the issuer cache as requiring sign-in. The issuer preserves the previous
encrypted bytes, permits cleanup of that exact worker, and does not reissue the
rejected cache. The next attempt uses the existing browser authentication Human
before allocating a VM; completing sign-in resumes it with the retained plan.
A missing, locked or partially corrupt cache still fails closed. Cleanup retries
use `cleanup_error` and never overwrite the recorded execution failure.

## Retained input dependencies

Inputs/tooling require a consistent Broker, gateway, Runner and bootstrap set.
Drain v2 runs before rolling back binaries; older builds can discard new fields
when reading state. Restore a prior database only as part of an explicitly chosen
consistent recovery point, accounting for all admissions after that point.

Use `execution_start.inputs` only with the exact source run, artifact name and
SHA256, in the original chat/topic. The source must have completed cleanup and
still be retained when admitting the dependency. Retention automatically holds
its bytes until dependent cleanup or the original dependent deadline plus five
minutes; restart does not extend this limit. A canceled admission takes no hold.

`artifact_input_failed` includes the concrete failed transfer prerequisite in the
terminal detail. Changed ETag/bytes, inaccessible sources, invalid ranges and
redirects stop preparation before Callee starts. A lost Runner process may resume
only its explicit input preparation checkpoint. Do not relabel a lost active
Callee process as an input checkpoint to force a replay.

The internal `/worker-inputs/{run_id}/{name}` route accepts only the scoped,
expiring worker token and supports byte ranges. It is not a public artifact link.
Keep the normal Broker token out of workers. Input preparation does not install
tools, run an archive's contents, or expand a managed-host approval.