Files
obsidian-vault/personal/projects/archive/execution-broker-repo-cleanup-2026-10-16/docs/recovery.md
T

149 lines
8.9 KiB
Markdown
Executable File

# Архив исходника Broker
Историческая копия до очистки репозитория 2026-10-16. Источник: `mallexxx/execution-broker`, commit `ae92c15d31c8396789765cebf82f7971d6f4cb12`, файл `docs/recovery.md`. Старые статусы и deployment pins ниже не заменяют актуальную [[personal/projects/balda/execution-broker|эксплуатационную карточку]].
---
# Stack recovery
`scripts/stack-recovery.py` takes a coordinated **offline** snapshot, verifies it,
and restores into a new inert directory. Python 3.12+ and Docker CLI are required.
The plan is an operator-owned file, never model/tool input. Schedule a maintenance
window and finish all runs/cleanup before backup. Balda stops first; Broker,
Prism and gateway stop next. Only previously running containers restart, in
reverse order, even when capture fails or receives SIGINT/SIGTERM. A forced
shutdown, live Docker worker, credential owner, pending Broker run, unsafe tar
member or corrupt SQLite database prevents publication of `manifest.json`.
Example plan (replace every path/container with the installed deployment):
```json
{
"containers": ["balda", "broker", "prism", "worker"],
"vault_dir": "/private/operator/credentials",
"credential_status_command": ["/opt/execution-broker/execution-credentials", "--operation", "status", "--vault", "/private/operator/credentials"],
"sources": [
{"name": "balda-state", "container": "balda", "path": "/state"},
{"name": "broker-state", "container": "broker", "path": "/state"},
{"name": "prism-state", "container": "prism", "path": "/state"},
{"name": "beads-state", "container": "prism", "path": "/beads"},
{"name": "worker-state", "container": "worker", "path": "/state"},
{"name": "broker-config", "path": "/private/operator/broker-deployment"},
{"name": "balda-config", "path": "/private/operator/balda/config"},
{"name": "balda-environment", "path": "/private/operator/balda/.env"},
{"name": "balda-compose", "path": "/private/operator/balda/compose.yaml"},
{"name": "balda-secrets", "path": "/private/operator/balda/secrets"},
{"name": "credential-vault", "path": "/private/operator/credentials"}
]
}
```
Include service definitions, provider configuration, bootstrap assets and all
secret files referenced by deployment. Capture Virfield configuration, golden
images and signing-key escrow using their respective operator procedures; this
script does not create a disk/image snapshot of the VM manager. The credential
status command must access the same vault that `vault_dir` locks. Only this
Broker may issue credentials during the maintenance window. No other process may
write the listed state/configuration directories during capture.
```sh
mkdir -m 700 /private/backups
python3 scripts/stack-recovery.py backup --plan /private/recovery-plan.json \
--out /private/backups/stack-snapshot
python3 scripts/stack-recovery.py verify --snapshot /private/backups/stack-snapshot
python3 scripts/stack-recovery.py restore --snapshot /private/backups/stack-snapshot \
--out /private/restore-rehearsal
```
A snapshot is complete only when its manifest exists. Archives and manifest are
fsynced, files are private, and SHA-256 catches accidental corruption. This is
**not encrypted or signed backup storage**: SQLite/history/configuration may
contain plaintext secrets. Keep the private directory on encrypted storage and
use authenticated encrypted off-host backup. Preserve image digests in the
manifest and the corresponding distribution/images separately. A partial backup
is retained for diagnosis and never accepted by `verify`.
Restore refuses existing destinations and never starts services, contacts
Mattermost, issues credentials, or attaches production volumes. SQLite checks
include WAL recovery and foreign-key integrity. Archive links, path traversal,
absolute paths and special files are rejected. A failed restore has no
`RESTORE-INERT.json`; use a fresh destination for retry.
## Rehearsal and promotion
1. Restore into a new private directory. Start copied Broker with **no workers or
notifications**, new test bearer tokens and `--network none`. Share its network
namespace only with the copied Prism/probe containers. Read the accepted run
and verify every artifact's size/SHA-256 through the real HTTP API.
2. Start copied Prism with copied Beads/state and the isolated Broker. Replay an
existing Story request and run synchronization twice; Story identity and
fact count must stay unchanged. Check closed Story state through real `bd`.
3. Start copied Balda with external MCP/schedulers/webhooks disabled and a local
Mattermost protocol fixture in the same isolated network. Balda requires at
least one channel. Never point this rehearsal at the real Mattermost server.
Confirm startup, persisted sessions and NATS recovery, then stop the copies.
4. Read the copied vault with `execution-credentials --vault COPY --operation
status`. The original Keychain master key must remain available. Master keys
are deliberately absent from these archives: recovery on another machine
needs separately protected key escrow or fresh OAuth enrollment. Do not
overwrite a newer live OAuth cache with an old refresh token from backup.
5. Promotion is a separate operator operation: fence the old deployment and all
workers, reconcile resource/credential ownership, restore image/configuration
pins, verify routes, then enable one writer per datastore. Never run old and
restored control planes simultaneously against the same external resources.
Record the exact deployed image pins, snapshot hashes, database checks, artifact
hashes, credential generations and worker ownership in private operator evidence.
Keep these outcomes separate: drained same-host restore, cross-host synthetic vault
recovery, and restoration of actual working accounts after host or Keychain loss.
The first two do not establish the third. Production recovery requires an
independent protected master-key copy or a selected secret store; never store its
only recovery credential on the host being backed up. Restore the original keys
first, then attempt provider refresh. If protection material cannot be recovered
or the provider refuses refresh, preserve the old ciphertext and use fresh OAuth
after fencing old workers. A successful browser login must be followed by bounded
real-provider inference, checkpoint and cleanup before resuming ordinary work.
## Protected master-key escrow
`scripts/key-escrow.py` encrypts the enrolled identities' original master keys
to an exact GPG recipient fingerprint. Keep the recipient's private key in an
independent protected location and copy the encrypted package off-host. GPG is
required; recipient lookup is local and never downloads a key. Use the private
plan's `vault_dir` and `credential_status_commands` from the stack backup.
Every enrolled identity must be listed and idle, with no pending login. Export
takes the shared vault lock, detects concurrent record changes and preserves
encrypted records. Plaintext keys are passed in memory, not argv, diagnostics or
temporary files. Existing outputs are never overwritten.
```sh
python3 scripts/key-escrow.py export --plan /private/recovery-plan.json \
--recipient EXACT_RECOVERY_FINGERPRINT --out /private/backups/vault-keys.gpg
python3 scripts/key-escrow.py restore --bundle /private/backups/vault-keys.gpg \
--out /private/recovery-keys --gpg-home /private/independent-recovery-keyring
```
Restore creates private key-files and a metadata manifest in a new directory;
it starts no service and changes no existing vault. Validate each copied vault
with its restored `--key-file` before promotion. On Linux, configure the issuer
with that private key-file. On macOS, after validation, run
`python3 scripts/key-escrow.py install-keychain --keys /private/recovery-keys`.
Identical existing keys are retained; different keys cause a refusal. Keychain
installation trusts `/usr/bin/security`, used by the issuer, and sends key
material through stdin rather than argv.
Automated tests use synthetic identities. On 2026-10-16 the operator also accepted
recovery of all three working provider identities on an independent physical
Linux host without the original Mac or Keychain. The independently protected GPG
recipient, encrypted original master keys, matching vault records and recovery
software were retained off-host. Each copied vault passed native
issue/checkpoint/release and an authenticated read-only provider check (HTTP 200).
Temporary plaintext key-files were removed; original working vaults and provider
authorizations were unchanged.
Deliberate upstream revocation and fresh browser login were excluded by the
operator from the accepted scope and remain unverified. The successful restore
does not guarantee that a provider will accept an older token or allow unattended
recovery. This tool never silently resets a missing-key vault or revokes a live
session; fresh OAuth remains the fenced fallback described above.