[2026-10-16] eagle: personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/operations.md personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/telegram-approvals.md

This commit is contained in:
Alexey Martemyanov committed 2026-10-16 17:24:28 +06:00
1 parent 00a2d211af
commit e98d8ba61f
2 files changed
+389

No files matched your search

@@ -0,0 +1,229 @@
# Архив исходника Broker
Историческая копия из `ae92c15` до очистки репозитория 2026-10-16; текущая установка — [[personal/projects/balda/execution-broker]].
---
# Operations and recovery
Use `executionctl --token-file … --operation tools` for authenticated MCP reachability and `execution_status` for a specific run. `/healthz` is liveness only. JSON logs include run IDs and retry causes, without configured tokens. Virfield maintenance/capacity may keep a run provisioning until its deadline; inspect both services rather than creating another idempotency key.
For the local Compose stack, `deploy/compose.dynamic-images.yaml` pins selected
image tags. Compare those tags with the running containers before claiming a
source change is deployed. A Git push does not rebuild or restart the stack.
For Broker-managed local Git projects, macOS bootstrap automatically scopes the
guest configuration to the run's exact registered repository and enables its
private SSH Git tunnel. It retains the reviewed package digest and workflow pin;
new project names do not require a worker archive rebuild or manual allowlist.
The installed marker covers both the immutable archive and the effective guest
configuration, so a retry repairs an old guest configuration before starting.
Broker checks package workflow/profile compatibility before VM acquisition.
Permanent worker start refusals, including `forbidden`, `workflow_integrity` and
`bootstrap_incompatible`, end the run as failed and notify the original Balda
conversation with diagnostics. Temporary infrastructure failures retain retry
backoff; Telegram approval routing does not suppress terminal errors.
Project tools use [operator-owned tooling profiles](tooling.md). An operator
`default_tooling` entry can preserve a worker's existing standard Go environment;
an explicit task selection pins another approved recipe. SDKs are size/SHA-checked
and installed only in the disposable run directory, with their observed versions
retained in `tooling.json`. Golden images and the operator host are not modified.
Make/C/C++, Python, Node and JVM use the same package/archive/system backends.
Enable the independent [repository cache](repository-cache.md) only for approved
sources. [Environment changes](environment-changes.md) require an operator-authored
safe checkpoint and continuation in the pinned workflow pack; arbitrary graphs
are not resumed by replay. Deploy the matching execution/v3 gateway, Runner and
bootstrap before enabling either policy. An environment-change terminal outcome
must complete old-worker/credential cleanup before its separately approved
replacement can allocate anything. Preserve its original deadline and artifacts.
New workers persist whether provider credentials could have been used, and
record a stopped fence only after the Callee process group settles. An unused
preparation lease preserves the issuer cache. Missing admission/process metadata,
a missing used cache or an unknown supervisor outcome leaves the exclusive owner
locked for recovery; an empty phase is not evidence that credentials were unused.
Inspect `credential_state_lost`, `process_group_unsettled` and `cleanup_error`
instead of starting another attempt with the same account.
If a run ends with `callee_failed`, first read `execution_diagnostics` using its
bound chat and topic IDs. On updated Broker and worker builds, `failure_detail`
includes the bounded final adapter failure explanation from a checksum-verified
`callee.log`, even when no `quality.json` was produced. Earlier transport failure
classes still distinguish a missing result, process exit or signal, malformed
event, or output read failure. Quality diagnostics retain review and verification
findings separately.
Check the last role event and the `callee.log` artifact to locate the boundary;
an empty log or generic `callee_failed` does not prove an agent started or
crashed. Balda must report the cause as unrecorded when diagnostics has neither
a quality finding nor `failure_detail`; it must not guess from an empty artifact.
For Broker-terminal failures such as `lease_failed`, the terminal code replaces
any earlier retry code in `last_error`. Historical runs that ended before this
fix can still show a stale retry code; inspect Virfield lease events and the
prepare job for the actual resource failure. Older deployed builds may not
expose `failure_detail`; update Broker
and the worker package before relying on it. The Callee adapter accepts an
approval question up to 256 KiB. Runner and the journal retain a complete Human
event up to 512 KiB after JSON serialization; ordinary event summaries retain
their 32 KiB limit. Runner event-processing errors are redacted and saved in
`callee.log` before artifact collection, so their concrete cause is available to
`execution_diagnostics`. A
confirmed old 16 KiB
prompt cap requires an updated VM bootstrap package before retrying with a new
Story/run key. A Story created under placeholder `0/0` cannot sync with the real
Balda Session ID; create a new Story and Task bound to the current Session ID.
## VM readiness failures
`guest_keychain_failed` is a Virfield preparation failure before the worker is
ready. Its public error covers native GUI/SSH key storage and unlocked,
unobstructed console checks; it does not identify a password, OAuth or provider
failure. A failed prerequisite can leave no Runner/Callee log or artifacts.
Correlate the Broker run with the exact Virfield lease, prepare job and events.
Preserve bounded private provisioning diagnostics before cleanup when available,
within the existing deadline; never keep a failed lease indefinitely for logs.
After release its private directory may already be gone. Retain the public
failure and confirmed release, and record the unavailable diagnostic scope.
The [support matrix](support.md#runtime-matrix) records the production macos27
reacceptance passed on 2026-10-16 and preserves the three earlier failed probes.
The dated [Virfield record](https://github.com/mallexxx/virfield/blob/a1460d3372bdfbb5cb19d633d41dd5d3fb3164e6/docs/VERIFICATION.md#broker-macos27-reacceptance)
defines the tested scope; exact run/job/lease references remain in the private
operator incident register. An idle pool, a catalog entry or a different
OS/profile's success does not qualify another deployment.
On macOS, `kill(-pgid, SIGKILL)` can return `EPERM` for a group containing only
zombies. Runner settles that case only after a separate bounded process
observation finds no executable group members. An observation error or a live
member retains the credential fence; do not clear ownership from the signal
error alone.
## Recovery guarantees
Broker stores admission, stable acquisition inputs, every transition and events in SQLite WAL with FULL synchronous writes. It holds a process lock on its data directory. Run one Broker process per database on a local reliable filesystem; SQLite on NFS or multiple independent replicas is unsupported. Newer database schema versions fail closed.
Restart Broker freely: it replays uncertain acquisition/start with the same identity, redelivers unacknowledged Human answers and retrieves missed events. A running Runner stays alive inside its worker when Broker restarts. If the Runner process itself dies, it records a diagnostic failed outcome rather than rerunning an unknown workflow.
For cancellation call `execution_cancel`, retain the run ID, and wait for terminal **and** `cleanup=done`. VM cleanup belongs to Virfield, including Broker downtime. Docker Runner enforces its deadline; gateway independently reaps owned workers one hour after their lease deadline. This grace period permits artifact recovery after an outage. Resource slots remain occupied until release is confirmed. Docker and host-runner retain durable revocation records, so a delayed start cannot recreate a released run. Virfield admission refusals are distinguished from unknown acquisition outcomes; canceling a proven refusal does not create a VM just to delete it. Never manually remove resources belonging to another run to make capacity.
Artifacts resume from a private on-disk prefix after transfer interruption and Broker restart; the complete file is SHA-verified and fsynced before terminal publication. References expire after seven days. An hourly bounded retention pass removes expired payload directories only after confirmed worker cleanup and expiry of every reference. It preserves event, checksum, Story and idempotency history. Historical SQLite/Beads journals remain durable; monitor and size these separately. Partial unpublished transfers are reclaimed seven days after the original execution deadline. Loss of disk space fails transactions and retries without reporting success. Execution stops at its original deadline. Broker and Virfield reserve at most five additional minutes to collect already-written results; provider execution is not extended. Worker-side collection has its own one-minute bound. A lost worker before artifact copy can leave the run awaiting recovery until that bound; missing artifacts are not silently fabricated.
Bootstrap archives in the deployment `assets` directory and local `.build`
outputs have no automatic retention pass. After every run pinned to an old
package has reached terminal state and `cleanup=done`, compare all configured
worker `bootstrap_file` paths with the archive directory before removing old
packages. Check `df` after cleanup: APFS clones and local Time Machine snapshots
can retain blocks even when `du` for the directory shrinks. Virfield reserves
the VM's full virtual disk size plus its host safety reserve at admission; a
run can be rejected while the image's currently allocated size is smaller.
## Backup and restore
Use `execution-broker --config … --backup /absolute/new-backup.db` for a consistent live SQLite snapshot. The destination must not exist. Copy the Broker artifact directory alongside the snapshot; pause new admission for a fully coordinated database/artifact backup. Transport keys are included in the private database: encrypt backups and restrict access to the service administrator.
For Prism/Beads backup, stop the Prism sidecar and take consistent snapshots of **both** `prism-state` and `beads-state` volumes, then start it again. The projection database retains facts/cursors and the Beads directory retains Dolt history; backing up only one is not a complete recovery point.
Restore into an empty data directory with the same original worker profiles, bearer secrets and approval public keys. Stop the existing instance first. Restore the database/artifacts with restricted permissions and restart one service. Verify pending run statuses and resource cleanup before accepting new work. Do not restore an old snapshot while the original instance is still accepting requests; lost post-snapshot idempotency history can duplicate external work.
## Balda delivery
Prism projection and chat delivery have independent cursors. An adapter ack means Beads facts were applied, not that a chat message was delivered. Optional Broker `notifications` routes call Balda's durable inbound webhook with an idempotency header; the destination must deduplicate that header. By default, a 2xx advances the notification cursor. With `require_delivery_ack=true`, a Balda `202` leaves the event pending and Broker retries the same key; only `200` with a durable provider receipt advances the cursor. The payload contains only run/Story/session/event references; Balda reads authoritative state through MCP before delivering a response. A route may opt into `coalesce_cleanup=true`: Broker persists whether cleanup was already `done` before the first terminal webhook attempt. Only that case permits advancing a later `cleanup_completed` event without another post. Cleanup that finishes during asynchronous delivery is notified separately, including after a Broker restart; a later receipt cannot retroactively change what the terminal message said. The default remains separate notifications.
The packaged Balda skill persists each attempt and arms a native original-session
one-shot wait before execution admission. It uses `execution_lookup` after a lost
Start/retry response and closes abandoned admissions atomically before declaring
that no execution can arrive later. Each wake arms a fresh generation before
reading Broker state; a separate TTL recovery wait covers a failed intermediate
model turn. Consult [Balda integration](../integrations/balda/README.md) for the
upgrade contract. A wait acknowledgement is not a chat-provider delivery receipt.
Monitor failed observer turns and retain run IDs for manual recovery.
Current generic Balda webhooks execute in isolated private jobs. Retired session
envelope fields are invalid; a wildcard route cannot continue arbitrary sessions.
Disable obsolete notification routes while preserving their cursors and audit
history. Do not mark retained notifications delivered as part of a migration.
Existing Balda sessions pin capability snapshots; a new conversation adopts the
upgraded skill without modifying an existing conversation's history.
## Story quality evidence
Implementation, verification and review run after the deterministic owner approval
gate. Their authored context identifies that approval as a workflow-journal fact,
not a repository file. Worker checkouts intentionally use detached HEAD at the
pinned starting revision; local commits are exported in `commits.bundle` and do
not require a named branch or an upstream push.
Every verifier check is a repeatable shell assertion with `exit_code=0` and
`expected_exit_code=0` (the default). A negative CLI behavior check wraps the
application invocation in an assertion of its exit code and output; the assertion
itself must succeed. The deterministic quality gate repeats each assertion in
the same disposable checkout and retains its actual exit and bounded output.
It never accepts a failed suite because the model declared that failure expected.
A mismatch, unexecuted check, unresolved
acceptance finding or malformed report fails the quality gate and prevents code
publication. Non-blocking observations belong in optional string `notes`.
## One worker per run and bounded repairs
A worker belongs to a run, and a terminal outcome releases it after artifact
collection/checkpoint. An unfinished Prism Story does not by itself retain a VM.
Coding workflows for Codex, Claude and OpenCode keep implementation, verification
and review in the same checkout for up to three repair iterations. Failed test
or review reports feed the next implementation session without a new run, VM or
plan-approval replay. The final quality gate still rejects an exhausted cycle.
An intentional new attempt with the same Story, originating chat/topic, task,
repository revision, workflow ID, worker, image and agent reuses the exact
approved plan text from its durable Human journal. It starts at the authored
implementation/repair/quality subtree rather than regenerating a plan. A new
idempotency key and repaired workflow pack do not invalidate that decision;
changed execution scope or later owner rejection/amendment requires approval.
Approval snapshots travel only in private worker manifests and are omitted from
public status. ACP permission and provider sign-in requests remain independent.
Infrastructure/process failures, explicit cancellation and the original deadline
remain terminal; they are not indefinite VM retention or blind workflow replay.
Runner and its separate RPC processes open the same SQLite journal. Write
transactions reserve the writer before reading the update snapshot
(`BEGIN IMMEDIATE`), avoiding a deferred-transaction SQLITE_BUSY_SNAPSHOT (517).
The busy timeout is initialized per connection before normal database setup.
`execution_diagnostics` reads the checksum-verified Runner/Callee log for every
unsuccessful terminal state, including expired and canceled runs. For example,
an approval wait that reaches the original run deadline reports
`context deadline exceeded` rather than only `callee_event_processing_failed`.
Logs produced before Runner recorded the underlying error cannot reconstruct
that missing cause retrospectively.
If Claude clears every native OAuth token after an authentication rejection,
the stopped-worker checkpoint retains its account/organization binding and
marks the issuer cache as requiring sign-in. The issuer preserves the previous
encrypted bytes, permits cleanup of that exact worker, and does not reissue the
rejected cache. The next attempt uses the existing browser authentication Human
before allocating a VM; completing sign-in resumes it with the retained plan.
A missing, locked or partially corrupt cache still fails closed. Cleanup retries
use `cleanup_error` and never overwrite the recorded execution failure.
## Retained input dependencies
Inputs/tooling require a consistent Broker, gateway, Runner and bootstrap set.
Drain v2 runs before rolling back binaries; older builds can discard new fields
when reading state. Restore a prior database only as part of an explicitly chosen
consistent recovery point, accounting for all admissions after that point.
Use `execution_start.inputs` only with the exact source run, artifact name and
SHA256, in the original chat/topic. The source must have completed cleanup and
still be retained when admitting the dependency. Retention automatically holds
its bytes until dependent cleanup or the original dependent deadline plus five
minutes; restart does not extend this limit. A canceled admission takes no hold.
`artifact_input_failed` includes the concrete failed transfer prerequisite in the
terminal detail. Changed ETag/bytes, inaccessible sources, invalid ranges and
redirects stop preparation before Callee starts. A lost Runner process may resume
only its explicit input preparation checkpoint. Do not relabel a lost active
Callee process as an input checkpoint to force a replay.
The internal `/worker-inputs/{run_id}/{name}` route accepts only the scoped,
expiring worker token and supports byte ranges. It is not a public artifact link.
Keep the normal Broker token out of workers. Input preparation does not install
tools, run an archive's contents, or expand a managed-host approval.
@@ -0,0 +1,160 @@
# Архив исходника Broker
Историческая копия из `ae92c15` до очистки репозитория 2026-10-16; текущая установка — [[personal/projects/balda/execution-broker]].
---
# Telegram approval actions
A retry of the same Story and exact execution scope retains the actual plan
previously approved by the owner. Broker loads that decision from its journal
and sends a private, checksum-bound snapshot to Runner/Callee. Callee continues
implementation and the quality gate without regenerating planning or asking for
the same approval again. Changes to the task, repository revision, executor or
session require a new approval. Later owner feedback is never replaced with an
older approval. ACP permission choices and command authorization remain separate.
Broker can deliver pending workflow and worker ACP permission requests to a
dedicated Telegram bot, even when the run started in Mattermost. Balda's
existing notification route remains responsible for the original conversation
and terminal result. Telegram never creates a new Story, run or Balda session.
The original Balda conversation is the default approval channel. When Telegram
is enabled for a source session, approval requests are delivered only to Telegram;
Balda does not repeat the question or options. Result and cleanup notifications
continue in the original conversation. The existing command authorization API
remains available for operator recovery. Authored command allow/deny/ask
policies, bearer authentication and operator grants retain their existing roles.
## Configure
Create a dedicated bot with BotFather. Save its token in the deployment's private
`telegram-approval.token` file, mode 0600; never put it in Git or a plugin.
Set the owner's numeric Telegram user ID from an independently established
identity. The bot does not enroll the first user who contacts it. The owner
must open the bot's private chat and press Start before it can deliver requests.
Add this operator-owned block to `broker.json`:
```json
{
"telegram_approvals": {
"enabled": true,
"token_file": "/run/secrets/telegram_approval_token",
"owner_user_id": 123456789,
"chat_ids": ["mm-c-EXACT_SOURCE_SESSION"],
"timeout_seconds": 1800,
"min_run_ttl_seconds": 3600
}
}
```
Omit this block or set `"enabled": false` to use the current approval method
without starting a Telegram client, reading its token or requiring a bot secret.
Only `"enabled": true` selects the Telegram approval route. The Bot API transport
does not replace command authorization or change the original approval API.
This example allows 30 minutes for a Telegram decision and requires at least one
hour of total run TTL, leaving time for planning and execution. For selected
source sessions, `min_run_ttl_seconds` rejects a new shorter `execution_start`
before admission or worker allocation; retry the same unaccepted request with
an adequate `ttl_seconds`. Zero or omission preserves the caller-selected TTL.
Already admitted requests retain their original deadline and idempotent replay.
The execution deadline remains a hard bound, including queueing and planning;
it is never extended when a question appears or the owner refreshes controls.
`chat_ids` identifies original Balda sessions. An explicit `"*"` selects all
sessions in this single-operator Broker. It does not enable multi-user approval.
Include `deploy/compose.telegram-approvals.yaml`, rebuild the Broker image,
select that reviewed image in an operator override, and recreate only Broker
after checking current runs and retaining a rollback image/configuration.
The default deployment image pins remain unchanged. The bot uses outgoing long
polling; no new public listener is needed. A configured Telegram webhook prevents
polling and is never removed
automatically. Run only one consumer for this bot token.
## Actions and ownership
- Workflow approval offers Approve and Cancel. Reply to the exact prompt to amend
the requirements where the selected workflow supports amendments.
- Updated Callee adapters send structured ACP option IDs and labels. The buttons
offer only the provider's actual choices, such as Allow Once and Deny. Authored
`allow` and `deny` roles retain their existing policy; only `ask` requests pause.
Update worker packages as well as Broker to obtain structured ACP actions.
- Every prompt also offers Cancel run. Unknown or older generic questions accept
a reply to their exact Telegram message instead of inventing an approval value.
Known installed Story packages retain their existing approved/cancel contract.
- Only the configured owner's user ID in that owner's private chat can decide.
Callback data contains an unpredictable, durable request nonce and an option
index. It never contains credentials, paths, shell commands or approval answers.
- Broker submits the selected answer against the original run, Human request,
chat and topic. The accepted-response journal records the Telegram user ID.
Concurrent operator recovery and Telegram responses accept one decision; a
stale cancellation button cannot cancel an already answered request.
Each request sends one readable message with its action buttons. Workflow plans
use the task's language and omit internal Story/run/request IDs. A long question
is explicitly marked as a preview and offers **Full details** (Russian: **Полный
план**). Only clicking that button sends the complete UTF-8 `approval.txt` file;
it does not approve, cancel or resume the worker. Repeated clicks reuse the
recorded document delivery. Read the complete details before approving a preview.
Reply to the message with buttons to change a Story plan; a standalone message
or a reply to its attachment is not an approval response. Up to three requirement
amendments regenerate planning in the same run and ask for approval again.
ACP permission prompts accept only their exact offered option IDs; text feedback
is not a requirement amendment for those prompts.
## Recovery and expiration
Prompt bindings, delivery IDs and deadlines live in the private Broker run state.
The Bot API update cursor is atomically saved in the Broker data directory after
processing an action. A prompt is sent without buttons, its message ID is saved,
and only then are controls attached. Restart completes an interrupted control
publication on that same message. An ambiguous send may leave a duplicate text
message without buttons; it cannot leave an untracked actionable prompt.
Duplicate callbacks do not resume a worker twice. Resolved controls are removed;
clicking an older actionable copy also removes that copy's controls before the
update is acknowledged. Already expired callback acknowledgements do not block
later updates.
The Telegram timeout is bounded by the original execution deadline. Expiry removes
the approval controls and rejects their late actions. The same message explains
expiration and offers **Refresh request**. Only the configured owner can refresh
a still-pending Human; delivery creates a fresh nonce and asks for a new decision
in the same run and worker. The renewal request is saved before clearing the old
controls; after a restart, delivery finishes renewal without another callback.
It does not accept an approval, extend the execution
deadline, or send the request into Mattermost. Stale callbacks, changed Human
requests and expired executions cannot refresh it. The existing command API
also remains available until the normal run deadline. Telegram
failures or expiry do not send a duplicate question into the original conversation.
Disabling Telegram restores conversation delivery for a still-pending request.
An explicit Cancel run button still requests cancellation.
Replies to stale prompts, other users, group messages,
unknown actions and changed request bindings cannot resume execution.
Provider sign-in challenges are excluded from this route. Managed-host signed
capabilities and folder-access operator grants retain their independent
authorization paths. This feature covers Broker-managed runs; Balda's own host
ACP permission service is a separate integration. The context-review subagent
and policy-based autoapproval are separate planned work.
## Verification
Run `make check` and `scripts/check-callee.sh`. Tests cover allow/deny actions,
exclusive approval delivery with Telegram enabled or disabled, bot failures,
active status/event polling, switching a pending request back to the conversation,
and continued terminal/result delivery, as well as
the exact worker response, competing channels, unauthorized and stale clicks,
timeout, restart, ambiguous sends, interrupted control publication and renewal,
stale-copy cleanup failures, acknowledgement failure, complete documents,
requirement amendments, secret-safe transport errors and private API projection.
On 2026-10-12, candidate `05358d0` passed an isolated Docker acceptance run
with the real Approval bot. The configured owner clicked Approve; Broker recorded
one attributed response, the deterministic Callee fixture succeeded, all five
artifact hashes matched, and worker container/volume cleanup was independently
verified. The test stack had no Balda notification route, so this does not prove
Mattermost delivery or a real-provider ACP permission request. The production
rollout and source-session scope require their own deployment verification.
Bot transport contracts: [Telegram Bot API](https://core.telegram.org/bots/api).