Backup and recovery

Better Compute keeps almost everything in one PostgreSQL database, a few things in files that only operators hold, and a small amount of state on every node. This page lists each piece, what losing it costs, and how to get back.

What lives where

On the control plane

Held by people, outside the deployment

On each node

Under bcd's data_dir (/var/lib/bcd by default) and /etc/bcd; the full layout is in files on the host.

Backing up

PostgreSQL

Use PostgreSQL's own tools. Nothing in Better Compute adds to or replaces them.

Recommendation.

• Continuous archiving with point-in-time recovery (base backups plus WAL), so you can choose the restore point and lose as little as possible. A nightly pg_dump alone loses up to a day, and a day matters here because node certificates turn over every 14 hours or so.
• Back up the whole database, every schema and table: compute's base schema and every module's tables share it, and compute_migrations must stay consistent with them.
• Keep the backups away from the database host, and test a restore on a scratch server regularly. better-compute doctor against the restored copy is a quick sanity check: it reports a database that was never migrated, missing migrations of this release, a missing btree_gist, and a compute_now() that does not read the database clock.
• Back up Better IAM's store at the same moment, so tenants and grants match the compute data that refers to them.

Two things in the database need care beyond that:

• Detached history partitions. The retention module and the audit archiving step can detach monthly partitions of audit_events, inventory_snapshots and usage_windows; a detached partition is a standalone table you export and drop yourself. Keep those exports with your backups. See archiving and history partitions.
• An audit copy outside the database. An audit sink copies every committed audit_events row out in commit order (see audit sinks). It is the record that survives a database restore.

The PKI bundle and the secret

Back up the bundle and BETTER_COMPUTE_SECRET together, and keep the offline root key apart from both. The bundle file is sealed: its private keys are encrypted under a key derived from the secret, and the seal also covers every certificate and pin, so a copy without the secret is useless and a copy changed without the secret no longer loads (INVALID_CONFIG: "restore it from a backup"). See PKI and the deployment secret.

Take a new copy whenever the bundle changes: pki rotate-command-root, pki renew-ca, pki add-sandbox-ca and pki reseal all write it back. A bundle restored from before a command-signing root rotation lacks the new root, and daemons that already trust only the new one would refuse every command signed under the old.

Nodes

Recommendation. Do not back up bcd's state files to restore them later. Recover a node by joining it again instead (below). Restoring an old copy is worse than losing the file:

• an old journal.db keeps its journal_id, so the control plane believes the journal is intact and replays in-flight commands into a journal that no longer knows they ran: the journal-lost rule, which exists to stop a second execution, would not fire;
• old batch-leaves.json and unbatched-destructive.json forget leaves and slots already used, so an approved batch leaf could run again;
• an old update.json lowers the release floor that keeps the node from going back to an older binary.

Tenants' volumes and instance disks are the exception: they are tenant data, and the code does not replicate them. Whether and how to back them up is a decision for your service; nothing in bcd snapshots a volume.

Restore order

better-compute migrate

better-compute nodes cordon <node> --tenant <tenant> --reason 'database restored'

What a restore cannot bring back

A restore rewinds the database, but not the nodes, the operators' keys or the outside world. Everything below follows from how the code treats state it does not recognise.

Work done after the backup

Reservations, commands, workflows, instances and audit events written after the backup are gone from the control plane. A daemon still holding unacked results replays them at its next Hello; events it had already had acknowledged were deleted from its journal and are lost. Anyone holding an audit checkpoint signed after the backup can tell the trail was rewound (see consistency proofs).

Desired state ahead of the database

bcd applies a desired state only when its generation is above the one it last applied (daemon/internal/agent/spec.go), and a gateway sends a spec only when the node's applied generation is below the database's. A node that applied generations written after the backup therefore ignores the restored specs and keeps enforcing its newer one, its holds, instances, volumes and access included, until the control plane's generation for it passes the node's. The specs that follow are built from the restored state, and the instance converger treats its section as the whole desired set: an instance created after the backup is removed, and a volume the section no longer lists and no running VM has open is destroyed, punched and unlinked (see persistent volumes). Nothing in the code detects or warns about this gap.

Recommendation: keep the node cordoned and compare its observed_generation and what it runs (bcd status) with the restored records before letting new specs reach it.

Certificates issued after the backup

Node certificates live 24 hours and are renewed at about 60% of that, so most nodes' current certificate is younger than a backup taken more than about 14 hours earlier. Such a certificate is missing from the restored node_certs:

The node keeps working until its certificate expires, then loses its session. Rejoin it before that (below). Nodes that joined for the first time after the backup have no row at all and are refused at Hello; they join again as new machines, or reclaim their old id if the restored database still holds their hardware key.

Revocations and consumed tokens

Revocations made after the backup are undone: a certificate you revoked is valid again in the restored node_certs. Re-run them (nodes.revokeIdentity; see revoking a node's identity). Join tokens used after the backup have their uses back, and tokens revoked after it are live again: revoke every token that should not be usable (better-compute join-tokens list, then join-tokens revoke).

The trust bundle version

A daemon takes a trust bundle only when its version is above the one it holds. The restored trust_bundle_state may be below what daemons already hold (for example after a command-signing root rotation that happened after the backup), and then the bundle the restored deployment publishes is ignored. pki.trustVersion is a floor: set it at or above the highest node_sessions.trust_version the reconnected daemons report, and the next published bundle reaches them. better-compute pki trust-version shows how many connected daemons are behind. See signed intent.

Releases

A release registered after the backup is missing from daemon_releases. Nodes that confirmed it hold a release floor at its sequence and refuse every older binary, so a rollout of an older registered release fails on them. Register the newer manifest again; releases.register accepts it because its sequence is above every registered one.

Rejoining a node

The same steps cover a node whose disk was lost, a node whose certificate the restored database does not know, and a node whose certificate expired while it was offline.

1. A certificate expired less than 7 days ago, with the x509pop attestor configured and the certificate in node_certs, re-joins with its old key. See renewal.

2. Otherwise, move the old identity aside and join again, as root:

   bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt     # with a token
   bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm       # or with the TPM

   --reset renames identity.json, node.key, a pending node.key.next and journal.db to <name>.reset-<UTC time> and deletes nothing. See joining again with --reset.

3. Same node or a new one. A machine whose trusted hardware key the database already holds reclaims its node id with a token that has allowReclaim (identity generation up by one, older certificates revoked); a tpm join with the same endorsement key reclaims the node bound to it. Without either, delete the old node first and the machine joins as a new one. See placeholder hardware and reclaims.

4. A lost disk also lost config.toml and the binary: re-run the install script with a token, which installs, joins and starts bcd in one go.

The new journal makes the control plane fail the node's in-flight non-retry-safe commands with journal-lost, and their owners re-decide; see resume rules.

Disaster scenarios

The PKI bundle or the secret is lost

There is no recovery that keeps the fleet. A bundle that is lost, or no longer opens because the secret changed, cannot be recreated: pki init makes new CAs and new JoinService keys, so every gateway certificate, every node certificate and every pin changes.

1. better-compute pki init with the same trust domain, and pki issue-gateway for every gateway.
2. New install scripts (daemon install-script reads the new pins) and join tokens.
3. On every node, as root: re-run the new script, then bcd join --reset --yes with a token. With the database intact, machines with a trusted hardware key reclaim their node ids through tokens with allowReclaim.
4. Every secret in the default sealed store is unreadable if the secret, not only the bundle, was lost: set them again.

A release signing key is lost

Create new keys with release keygen, put their pins in rolloutModule({ release: { pins } }), and give every node the new pins in /etc/bcd/config.toml under [update] release_pins. Re-running the install script does not do it: when config.toml already has an [update] section, the script keeps it and prints that --release-pins was not applied. Until a node has the new pins, it refuses every release signed by the new keys. Sign the next release with a sequence above the last one, since nodes keep their release floor.

Better IAM

Compute stores no IAM tenants, principals or policies of its own. It keeps the authorization projection (authz_projection, with its change feed authz_changes) that IAM's events maintain, and iam_inbox, the events it still has to apply. Restoring compute and IAM to different points leaves the projection describing one and the resources the other; recommendation: back both up at the same moment and restore both to it.