8.9 KiB
Executable File
Архив исходника Broker
Историческая копия до очистки репозитория 2026-10-16. Источник: mallexxx/execution-broker, commit ae92c15d31c8396789765cebf82f7971d6f4cb12, файл docs/recovery.md. Старые статусы и deployment pins ниже не заменяют актуальную personal/projects/balda/execution-broker.
Stack recovery
scripts/stack-recovery.py takes a coordinated offline snapshot, verifies it,
and restores into a new inert directory. Python 3.12+ and Docker CLI are required.
The plan is an operator-owned file, never model/tool input. Schedule a maintenance
window and finish all runs/cleanup before backup. Balda stops first; Broker,
Prism and gateway stop next. Only previously running containers restart, in
reverse order, even when capture fails or receives SIGINT/SIGTERM. A forced
shutdown, live Docker worker, credential owner, pending Broker run, unsafe tar
member or corrupt SQLite database prevents publication of manifest.json.
Example plan (replace every path/container with the installed deployment):
{
"containers": ["balda", "broker", "prism", "worker"],
"vault_dir": "/private/operator/credentials",
"credential_status_command": ["/opt/execution-broker/execution-credentials", "--operation", "status", "--vault", "/private/operator/credentials"],
"sources": [
{"name": "balda-state", "container": "balda", "path": "/state"},
{"name": "broker-state", "container": "broker", "path": "/state"},
{"name": "prism-state", "container": "prism", "path": "/state"},
{"name": "beads-state", "container": "prism", "path": "/beads"},
{"name": "worker-state", "container": "worker", "path": "/state"},
{"name": "broker-config", "path": "/private/operator/broker-deployment"},
{"name": "balda-config", "path": "/private/operator/balda/config"},
{"name": "balda-environment", "path": "/private/operator/balda/.env"},
{"name": "balda-compose", "path": "/private/operator/balda/compose.yaml"},
{"name": "balda-secrets", "path": "/private/operator/balda/secrets"},
{"name": "credential-vault", "path": "/private/operator/credentials"}
]
}
Include service definitions, provider configuration, bootstrap assets and all
secret files referenced by deployment. Capture Virfield configuration, golden
images and signing-key escrow using their respective operator procedures; this
script does not create a disk/image snapshot of the VM manager. The credential
status command must access the same vault that vault_dir locks. Only this
Broker may issue credentials during the maintenance window. No other process may
write the listed state/configuration directories during capture.
mkdir -m 700 /private/backups
python3 scripts/stack-recovery.py backup --plan /private/recovery-plan.json \
--out /private/backups/stack-snapshot
python3 scripts/stack-recovery.py verify --snapshot /private/backups/stack-snapshot
python3 scripts/stack-recovery.py restore --snapshot /private/backups/stack-snapshot \
--out /private/restore-rehearsal
A snapshot is complete only when its manifest exists. Archives and manifest are
fsynced, files are private, and SHA-256 catches accidental corruption. This is
not encrypted or signed backup storage: SQLite/history/configuration may
contain plaintext secrets. Keep the private directory on encrypted storage and
use authenticated encrypted off-host backup. Preserve image digests in the
manifest and the corresponding distribution/images separately. A partial backup
is retained for diagnosis and never accepted by verify.
Restore refuses existing destinations and never starts services, contacts
Mattermost, issues credentials, or attaches production volumes. SQLite checks
include WAL recovery and foreign-key integrity. Archive links, path traversal,
absolute paths and special files are rejected. A failed restore has no
RESTORE-INERT.json; use a fresh destination for retry.
Rehearsal and promotion
- Restore into a new private directory. Start copied Broker with no workers or
notifications, new test bearer tokens and
--network none. Share its network namespace only with the copied Prism/probe containers. Read the accepted run and verify every artifact's size/SHA-256 through the real HTTP API. - Start copied Prism with copied Beads/state and the isolated Broker. Replay an
existing Story request and run synchronization twice; Story identity and
fact count must stay unchanged. Check closed Story state through real
bd. - Start copied Balda with external MCP/schedulers/webhooks disabled and a local Mattermost protocol fixture in the same isolated network. Balda requires at least one channel. Never point this rehearsal at the real Mattermost server. Confirm startup, persisted sessions and NATS recovery, then stop the copies.
- Read the copied vault with
execution-credentials --vault COPY --operation status. The original Keychain master key must remain available. Master keys are deliberately absent from these archives: recovery on another machine needs separately protected key escrow or fresh OAuth enrollment. Do not overwrite a newer live OAuth cache with an old refresh token from backup. - Promotion is a separate operator operation: fence the old deployment and all workers, reconcile resource/credential ownership, restore image/configuration pins, verify routes, then enable one writer per datastore. Never run old and restored control planes simultaneously against the same external resources.
Record the exact deployed image pins, snapshot hashes, database checks, artifact hashes, credential generations and worker ownership in private operator evidence. Keep these outcomes separate: drained same-host restore, cross-host synthetic vault recovery, and restoration of actual working accounts after host or Keychain loss. The first two do not establish the third. Production recovery requires an independent protected master-key copy or a selected secret store; never store its only recovery credential on the host being backed up. Restore the original keys first, then attempt provider refresh. If protection material cannot be recovered or the provider refuses refresh, preserve the old ciphertext and use fresh OAuth after fencing old workers. A successful browser login must be followed by bounded real-provider inference, checkpoint and cleanup before resuming ordinary work.
Protected master-key escrow
scripts/key-escrow.py encrypts the enrolled identities' original master keys
to an exact GPG recipient fingerprint. Keep the recipient's private key in an
independent protected location and copy the encrypted package off-host. GPG is
required; recipient lookup is local and never downloads a key. Use the private
plan's vault_dir and credential_status_commands from the stack backup.
Every enrolled identity must be listed and idle, with no pending login. Export
takes the shared vault lock, detects concurrent record changes and preserves
encrypted records. Plaintext keys are passed in memory, not argv, diagnostics or
temporary files. Existing outputs are never overwritten.
python3 scripts/key-escrow.py export --plan /private/recovery-plan.json \
--recipient EXACT_RECOVERY_FINGERPRINT --out /private/backups/vault-keys.gpg
python3 scripts/key-escrow.py restore --bundle /private/backups/vault-keys.gpg \
--out /private/recovery-keys --gpg-home /private/independent-recovery-keyring
Restore creates private key-files and a metadata manifest in a new directory;
it starts no service and changes no existing vault. Validate each copied vault
with its restored --key-file before promotion. On Linux, configure the issuer
with that private key-file. On macOS, after validation, run
python3 scripts/key-escrow.py install-keychain --keys /private/recovery-keys.
Identical existing keys are retained; different keys cause a refusal. Keychain
installation trusts /usr/bin/security, used by the issuer, and sends key
material through stdin rather than argv.
Automated tests use synthetic identities. On 2026-10-16 the operator also accepted recovery of all three working provider identities on an independent physical Linux host without the original Mac or Keychain. The independently protected GPG recipient, encrypted original master keys, matching vault records and recovery software were retained off-host. Each copied vault passed native issue/checkpoint/release and an authenticated read-only provider check (HTTP 200). Temporary plaintext key-files were removed; original working vaults and provider authorizations were unchanged.
Deliberate upstream revocation and fresh browser login were excluded by the operator from the accepted scope and remain unverified. The successful restore does not guarantee that a provider will accept an older token or allow unattended recovery. This tool never silently resets a missing-key vault or revokes a live session; fresh OAuth remains the fenced fallback described above.