149 lines
8.9 KiB
Markdown
Executable File
149 lines
8.9 KiB
Markdown
Executable File
# Архив исходника Broker
|
|
|
|
Историческая копия до очистки репозитория 2026-10-16. Источник: `mallexxx/execution-broker`, commit `ae92c15d31c8396789765cebf82f7971d6f4cb12`, файл `docs/recovery.md`. Старые статусы и deployment pins ниже не заменяют актуальную [[personal/projects/balda/execution-broker|эксплуатационную карточку]].
|
|
|
|
---
|
|
|
|
# Stack recovery
|
|
|
|
`scripts/stack-recovery.py` takes a coordinated **offline** snapshot, verifies it,
|
|
and restores into a new inert directory. Python 3.12+ and Docker CLI are required.
|
|
The plan is an operator-owned file, never model/tool input. Schedule a maintenance
|
|
window and finish all runs/cleanup before backup. Balda stops first; Broker,
|
|
Prism and gateway stop next. Only previously running containers restart, in
|
|
reverse order, even when capture fails or receives SIGINT/SIGTERM. A forced
|
|
shutdown, live Docker worker, credential owner, pending Broker run, unsafe tar
|
|
member or corrupt SQLite database prevents publication of `manifest.json`.
|
|
|
|
Example plan (replace every path/container with the installed deployment):
|
|
|
|
```json
|
|
{
|
|
"containers": ["balda", "broker", "prism", "worker"],
|
|
"vault_dir": "/private/operator/credentials",
|
|
"credential_status_command": ["/opt/execution-broker/execution-credentials", "--operation", "status", "--vault", "/private/operator/credentials"],
|
|
"sources": [
|
|
{"name": "balda-state", "container": "balda", "path": "/state"},
|
|
{"name": "broker-state", "container": "broker", "path": "/state"},
|
|
{"name": "prism-state", "container": "prism", "path": "/state"},
|
|
{"name": "beads-state", "container": "prism", "path": "/beads"},
|
|
{"name": "worker-state", "container": "worker", "path": "/state"},
|
|
{"name": "broker-config", "path": "/private/operator/broker-deployment"},
|
|
{"name": "balda-config", "path": "/private/operator/balda/config"},
|
|
{"name": "balda-environment", "path": "/private/operator/balda/.env"},
|
|
{"name": "balda-compose", "path": "/private/operator/balda/compose.yaml"},
|
|
{"name": "balda-secrets", "path": "/private/operator/balda/secrets"},
|
|
{"name": "credential-vault", "path": "/private/operator/credentials"}
|
|
]
|
|
}
|
|
```
|
|
|
|
Include service definitions, provider configuration, bootstrap assets and all
|
|
secret files referenced by deployment. Capture Virfield configuration, golden
|
|
images and signing-key escrow using their respective operator procedures; this
|
|
script does not create a disk/image snapshot of the VM manager. The credential
|
|
status command must access the same vault that `vault_dir` locks. Only this
|
|
Broker may issue credentials during the maintenance window. No other process may
|
|
write the listed state/configuration directories during capture.
|
|
|
|
```sh
|
|
mkdir -m 700 /private/backups
|
|
python3 scripts/stack-recovery.py backup --plan /private/recovery-plan.json \
|
|
--out /private/backups/stack-snapshot
|
|
python3 scripts/stack-recovery.py verify --snapshot /private/backups/stack-snapshot
|
|
python3 scripts/stack-recovery.py restore --snapshot /private/backups/stack-snapshot \
|
|
--out /private/restore-rehearsal
|
|
```
|
|
|
|
A snapshot is complete only when its manifest exists. Archives and manifest are
|
|
fsynced, files are private, and SHA-256 catches accidental corruption. This is
|
|
**not encrypted or signed backup storage**: SQLite/history/configuration may
|
|
contain plaintext secrets. Keep the private directory on encrypted storage and
|
|
use authenticated encrypted off-host backup. Preserve image digests in the
|
|
manifest and the corresponding distribution/images separately. A partial backup
|
|
is retained for diagnosis and never accepted by `verify`.
|
|
|
|
Restore refuses existing destinations and never starts services, contacts
|
|
Mattermost, issues credentials, or attaches production volumes. SQLite checks
|
|
include WAL recovery and foreign-key integrity. Archive links, path traversal,
|
|
absolute paths and special files are rejected. A failed restore has no
|
|
`RESTORE-INERT.json`; use a fresh destination for retry.
|
|
|
|
## Rehearsal and promotion
|
|
|
|
1. Restore into a new private directory. Start copied Broker with **no workers or
|
|
notifications**, new test bearer tokens and `--network none`. Share its network
|
|
namespace only with the copied Prism/probe containers. Read the accepted run
|
|
and verify every artifact's size/SHA-256 through the real HTTP API.
|
|
2. Start copied Prism with copied Beads/state and the isolated Broker. Replay an
|
|
existing Story request and run synchronization twice; Story identity and
|
|
fact count must stay unchanged. Check closed Story state through real `bd`.
|
|
3. Start copied Balda with external MCP/schedulers/webhooks disabled and a local
|
|
Mattermost protocol fixture in the same isolated network. Balda requires at
|
|
least one channel. Never point this rehearsal at the real Mattermost server.
|
|
Confirm startup, persisted sessions and NATS recovery, then stop the copies.
|
|
4. Read the copied vault with `execution-credentials --vault COPY --operation
|
|
status`. The original Keychain master key must remain available. Master keys
|
|
are deliberately absent from these archives: recovery on another machine
|
|
needs separately protected key escrow or fresh OAuth enrollment. Do not
|
|
overwrite a newer live OAuth cache with an old refresh token from backup.
|
|
5. Promotion is a separate operator operation: fence the old deployment and all
|
|
workers, reconcile resource/credential ownership, restore image/configuration
|
|
pins, verify routes, then enable one writer per datastore. Never run old and
|
|
restored control planes simultaneously against the same external resources.
|
|
|
|
Record the exact deployed image pins, snapshot hashes, database checks, artifact
|
|
hashes, credential generations and worker ownership in private operator evidence.
|
|
Keep these outcomes separate: drained same-host restore, cross-host synthetic vault
|
|
recovery, and restoration of actual working accounts after host or Keychain loss.
|
|
The first two do not establish the third. Production recovery requires an
|
|
independent protected master-key copy or a selected secret store; never store its
|
|
only recovery credential on the host being backed up. Restore the original keys
|
|
first, then attempt provider refresh. If protection material cannot be recovered
|
|
or the provider refuses refresh, preserve the old ciphertext and use fresh OAuth
|
|
after fencing old workers. A successful browser login must be followed by bounded
|
|
real-provider inference, checkpoint and cleanup before resuming ordinary work.
|
|
|
|
## Protected master-key escrow
|
|
|
|
`scripts/key-escrow.py` encrypts the enrolled identities' original master keys
|
|
to an exact GPG recipient fingerprint. Keep the recipient's private key in an
|
|
independent protected location and copy the encrypted package off-host. GPG is
|
|
required; recipient lookup is local and never downloads a key. Use the private
|
|
plan's `vault_dir` and `credential_status_commands` from the stack backup.
|
|
Every enrolled identity must be listed and idle, with no pending login. Export
|
|
takes the shared vault lock, detects concurrent record changes and preserves
|
|
encrypted records. Plaintext keys are passed in memory, not argv, diagnostics or
|
|
temporary files. Existing outputs are never overwritten.
|
|
|
|
```sh
|
|
python3 scripts/key-escrow.py export --plan /private/recovery-plan.json \
|
|
--recipient EXACT_RECOVERY_FINGERPRINT --out /private/backups/vault-keys.gpg
|
|
python3 scripts/key-escrow.py restore --bundle /private/backups/vault-keys.gpg \
|
|
--out /private/recovery-keys --gpg-home /private/independent-recovery-keyring
|
|
```
|
|
|
|
Restore creates private key-files and a metadata manifest in a new directory;
|
|
it starts no service and changes no existing vault. Validate each copied vault
|
|
with its restored `--key-file` before promotion. On Linux, configure the issuer
|
|
with that private key-file. On macOS, after validation, run
|
|
`python3 scripts/key-escrow.py install-keychain --keys /private/recovery-keys`.
|
|
Identical existing keys are retained; different keys cause a refusal. Keychain
|
|
installation trusts `/usr/bin/security`, used by the issuer, and sends key
|
|
material through stdin rather than argv.
|
|
|
|
Automated tests use synthetic identities. On 2026-10-16 the operator also accepted
|
|
recovery of all three working provider identities on an independent physical
|
|
Linux host without the original Mac or Keychain. The independently protected GPG
|
|
recipient, encrypted original master keys, matching vault records and recovery
|
|
software were retained off-host. Each copied vault passed native
|
|
issue/checkpoint/release and an authenticated read-only provider check (HTTP 200).
|
|
Temporary plaintext key-files were removed; original working vaults and provider
|
|
authorizations were unchanged.
|
|
|
|
Deliberate upstream revocation and fresh browser login were excluded by the
|
|
operator from the accepted scope and remain unverified. The successful restore
|
|
does not guarantee that a provider will accept an older token or allow unattended
|
|
recovery. This tool never silently resets a missing-key vault or revokes a live
|
|
session; fresh OAuth remains the fenced fallback described above.
|