Failure modes

The daemon protocol is built on the assumption that everything fails: processes crash, links go half-open, databases fail over, clocks drift, journals get wiped. The design goal is that each failure costs a reconnect and a replay, never a duplicated side effect, a double booking, or a released allocation. This page lists what happens for each failure, what state is lost, and which invariant the behavior protects.

The invariants

PLAN.md §15 defines the fleet invariants. Six of them are checked mechanically after every simulated run, and a seventh after a simulated night, by pure functions in @better-compute/testing (checkI1, checkI2, checkI3, checkI4, checkI5, checkI7, checkI8):

I6 (latency) is part of the design but has no checker in this module; the real-time benchmarks measure it.

At a glance

Gateway crash

A gateway process dies: no disconnect writes, no lease release. Everything it held in memory is gone, and it held nothing else.

1. Each daemon sees its stream end and redials with full-jitter back-off (0 to 1 s, doubling to 60 s). The L4 load balancer sends it to a live gateway.
2. The new gateway's Hello transaction bumps the epoch and routes the node to itself. In-flight commands (dispatched, acked, running) move to the new epoch and are replayed in seq order, except those the daemon reports as running; its unacked results are applied first. Pending commands need no rewrite: dispatch is a join on node_sessions.gateway_id, which now names the new gateway.
3. The daemon's journal deduplicates every replay: a done command resends its result, a running one re-attaches. Nothing runs twice (I2).
4. Nodes that do not come back are handled by the Liveness controller: once the dead gateway's gateway:<id> lease expires (20 s), its nodes become disconnected, and only after the 40-second lease TTL plus a 120-second gateway-loss grace do they become unreachable. That transition adds a NoSchedule reason and a condition and releases nothing (I5).

Tunnels on the dead gateway close; the caller opens a new one. In the fleet simulator with three real gateways and the real control plane, killing one re-homed its nodes with a p99 of 1 second at 1,000 nodes and 13.25 seconds at 10,000, with no invariant violations (bench/multi-gateway). PLAN.md's v0.4 target is under 60 seconds with no unreachable marks and no duplicate executions.

Gateway cut off from PostgreSQL

A gateway that loses the database, or stalls, cannot renew its gateway:<id> lease. After half the lease TTL (10 seconds) without a successful renewal, or as soon as it finds another owner on the lease, it fences itself: it forgets its lease and closes every stream UNAVAILABLE. Its daemons redial elsewhere.

A gateway that does not know it is stale (a paused process that wakes up) is stopped by two fences:

• every routing transaction begins by re-reading its lease FOR SHARE; a lease that moved aborts the transaction (FencedError), which fences the gateway;
• routing and liveness writes carry AND session_epoch = $myEpoch. After the node's Hello elsewhere, the zombie's dispatch marks and disconnect writes match nothing, and its liveness flush reports those nodes as gone, so it closes their streams ABORTED. Dispatch holds the session row FOR SHARE, so it cannot race an epoch bump.

Results and events the zombie still ingests are facts and are accepted from any epoch, deduplicated as usual. At most one gateway routes to a node at a time. tests/sim/multi-gateway.test.ts runs a stalled gateway that comes back as a zombie holding sessions its nodes have left.

Daemon crash or node reboot

Daemon crash. systemd restarts bcd after 5 seconds. The new process has a new incarnation id, and journal.Recover settles what the old one left behind:

The rule that makes this safe: accepted is fsynced before any side effect starts. A command is either known to the journal before it could have done anything, or it never ran.

Node reboot. The kernel boot_id changes. A self-terminating command (node.reboot, boot.kexec, daemon.upgrade) journaled running{boot_id} immediately before its side effect, so on restart the daemon reports it SUCCEEDED with rebooted, and the server's Hello transaction completes any in-flight self-terminating command the same way when it sees the new boot_id or role. Neither side ever replays it. Files under /run are gone after a reboot; the devices converger rewrites missing env files for assignments it still holds.

Self-update. A daemon.upgrade restarts bcd without a reboot, so the boot id does not change; bcd decides it from upgrade.json instead (de9f380). The new binary reports SUCCEEDED once its first Welcome arrives. A binary that crashes or never gets a Welcome within 120 seconds is swapped back (under systemd a crash loop also triggers bcd-rollback.service), and the old binary reports FAILED with update-failed and never tries that version again. On a node under bcd helper, the helper's own watchdog also swaps back an update still unconfirmed 30 seconds after its deadline and restarts bcd.service, so a new binary that never gets as far as asking cannot keep the node on it (2d67b25). The restarted process decides by where the current link points, never by the version it claims to run, so a lying binary cannot mark the new version failed and disarm the watchdog (c139b9c).

A bad release that confirmed. The swap back covers only an update that never confirmed. Once the new binary confirms, the release floor rises to its sequence and bcd refuses every older binary, the one still in its other slot included, so neither the watchdog nor a rollout can take the node back (c139b9c). rollouts.abort with rollback returns only nodes still upgrading (8347a3d). A release that confirmed and then misbehaves is fixed by rolling forward: sign and register a newer release with a higher sequence and roll it out; see rollouts. See self-update.

The helper is down or refuses. A bcd node running unprivileged beside bcd helper (28dc2ae) keeps its session and journal when the helper is gone, but every operation it sends there (a CDI spec or env file, the sshd files, the nftables table, a GPU reset, a reboot, a self-update step) fails with an error that starts helper:, like the direct operation failing. Such an error is not terminal, so a devices or access section stays PROGRESSING and is retried on the next pass (the spec is not observed meanwhile), and a command such as gpu.reset or node.reboot fails as it would without root. A node and a helper of different protocol versions refuse each other's messages, so restart both after replacing the binary. On a node installed with --unprivileged, bcd-helper.service restarts the helper after 2 seconds (Restart=always), and bcd.service Requires= it, so stopping the helper stops the node too; see running bcd node unprivileged.

Network partition and half-open links

When a node's link is cut, both ends notice through silence. The gateway closes the session after three missed heartbeats (30 seconds) and writes an epoch-fenced disconnect; HTTP/2 PINGs every 20 seconds detect a dead TCP connection. A half-open connection, where one end died silently, ends the same way, and when the node comes back its new session supersedes the old one by epoch.

While partitioned, the daemon keeps enforcing its last desired state and keeps every device assignment: it never drops one on its own. Workloads keep running. It journals events (up to 64 MiB and 10,000 unacked; metric events are dropped oldest-first when full, results and transitions never) and redials with back-off.

On the control plane the node goes disconnected, then unreachable after the 40-second lease TTL, with a NoSchedule reason and the NodeUnreachable condition. Only nodes of provider and vps-host pools can become lost, and only when the pool sets policy.lostAfter; metal never does automatically. No step releases an allocation, a booking or a command (I5). Commands that pass not_after while the node is away are swept to expired, or rejected EXPIRED by the daemon if they were already delivered.

Journal loss

A ramdisk reboot, a reinstall or corruption loses the journal. The daemon starts a new one with a new journal_id; its identity stays with its key, so it reconnects as the same node. bcd moves a journal file that does not open aside (journal.db.corrupt-<unix time>) rather than deleting it, and starts its server time mark from the copy in identity.json, so a lost journal does not reopen the clock-rewind window.

• The server sees the new journal_id in Hello and fails every in-flight command that is not retry-safe with reason journal-lost, instead of replaying it into a journal that cannot deduplicate. Controllers re-decide (I2).
• Retry-safe in-flight commands are replayed and may run again, which their kind allows.
• The event ack restarts at 0 for the new journal. Events still unacked in the lost journal are gone.
• The new journal holds no spec, so the daemon reports applied generation 0 and receives a full DesiredState at Hello.

The fleet simulator's journalLoss chaos and sim.loseJournal(nodeId) exercise this path.

PostgreSQL unavailable

PostgreSQL is the only source of truth, so an outage stops the fleet from changing without losing anything.

• Gateways cannot renew their leases and fence themselves within 10 seconds, dropping every stream. Without a lease they refuse new sessions UNAVAILABLE. A frame that fails to write also drops its stream. The lease loop keeps retrying, and once it renews, the gateway admits sessions again.
• Daemons back off up to 60 seconds between attempts, keep enforcing their last spec, and keep results and events in the journal until a gateway acknowledges them. Nothing is buffered in the gateway beyond its memory.
• Commands stay pending; nothing is dispatched. After the outage, those past not_after are swept to expired, and a stale command that does reach a node is rejected on not_after or min_generation.
• Liveness. When the Liveness controller finds a gap of more than 20 seconds since its last sweep, it holds every timed transition back for 5 minutes, so the ages the gateways could not flush do not read as dead nodes.

PLAN.md sizes this for a database failover of about 30 seconds: the fleet reconnects, nothing is marked unreachable, and nothing is released.

api or worker crash

api. api processes are stateless; a crash fails the requests in flight, and clients retry with their idempotency keys. Join, Renew and secret requests reach JoinService and the secret broker through the gateway's GatewayServices port. While the process serving them is down, Join and Renew fail and the daemon retries (a node certificate lasts 24 hours and renews at about 60% of its life, which leaves hours of margin), and secret requests are answered with the error unavailable.

Worker. Workers run the reconcilers, the CommandSweeper and the Liveness controller under table leases (15 seconds for shards, renewed every 5). When a worker dies, its leases expire and another worker takes them over; the work queue loses no wake-up because a claim that is not completed simply becomes claimable again. While no worker runs, gateways keep dispatching commands that are already pending and keep ingesting results, but nothing new is decided and nothing is swept. The Liveness controller's outage grace (above) applies when it comes back. See leases and fencing.

Clock skew

No component trusts another's wall clock.

• Gateways report ages. Liveness is flushed as "seen age_ms ago", and PostgreSQL computes compute_now() - age. A gateway whose clock is off cannot make a node look alive or dead (I4).
• Daemons use server time. Welcome.server_time gives the daemon the offset between its clock and compute_now(), and it evaluates not_after through that offset. Conditions carry age_ms on the daemon's monotonic clock, and the control plane derives since itself; event and inventory timestamps are advisory. Because server_time is unsigned, the daemon keeps a server time mark that only moves forward and refuses a Welcome more than 5 minutes before it, so a gateway cannot rewind the daemon into an expired signer's window (see Welcome).
• Control-plane processes anchor their business clock to compute_now(), re-sampled at every lease renewal. A process more than 1 second off raises ClockSkew: a log warning and a selfCheck finding, and the clock_offset_seconds metric shows the offset.
• Proofs tolerate 60 seconds: Renew proofs of possession, WebSocket proofs.

Certificate expiry

Node certificates last at most 24 hours (ramdisk certificates at most 2), and bcd renews at about 60% of the lifetime with Renew, proving possession of its current key. A session whose certificate expires is closed UNAUTHENTICATED ("certificate expired; renew and reconnect") by the gateway's liveness loop.

A certificate that expired while the machine was offline cannot be renewed. From v0.2, when the server has the x509pop attestor configured, the machine re-joins with its old key within 7 days unless the certificate was revoked. See renewal. Gateway serving certificates last 30 days in the PLAN.md design, which lists serving-certificate expiry among the fleet checks of doctor.

TPM cleared or unavailable

A node that joined with the tpm attestor and a certified node key has a certificate naming its TPM's attestation key (urn:bc:tpm-ak:), and since d94f1f3 it renews only with a next key that AK certifies. bcd never falls back to a file key for such a certificate:

• The TPM cannot be reached (a missing /dev/tpmrm0, or, for an unprivileged bcd node under [helper], a bcd helper that is down or cannot open the TPM: since 8870b8d that node reaches the TPM only through the helper). The renewal fails before any request: "the certificate's node key is in the TPM, which is unavailable" when bcd opens the TPM itself, or renewal: with the helper's error. bcd logs renewal failed with the attempt and expiresIn, and retries with backoff from 30 seconds to 10 minutes. Once the TPM answers again, the next retry renews. A TPM-held current key needs the TPM to load too, so a restarted bcd node does not start without it.
• The TPM derives another AK (another TPM, or one whose endorsement seed changed). The renewal fails the same way, with "this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join" (a1c1a46), and retrying cannot help.
• The TPM was cleared (TPM2_Clear). A clear replaces the owner hierarchy's seed but not the endorsement one, so the AK survives and the node key does not: the key is a primary of the owner hierarchy, the stored record derives another key, and bcd refuses its identity with "the node key does not match the node certificate: the TPM derives another key from node.key (was the TPM cleared?); join again: bcd join --reset --yes, then bcd join". Since a1c1a46 the cleared-TPM hint is here, where a clear shows, rather than on the AK check. bcd doctor fails its identity check with the same text.

Either way the certificate runs out within its 24 hours and the session closes as in certificate expiry. The recovery is to join again with the TPM, as root: bcd join --reset --yes --attestor tpm moves the old identity and journal aside (renamed <name>.reset-<UTC time>, not deleted) and joins, and the attestor reclaims the node bound to the machine's EK, with a new identity generation, a new certified key and a fresh AK statement. A machine whose EK changed (another TPM) joins as a new node instead, once the old node is cordoned and deleted (better-compute nodes delete <node> --yes, 4049621; nodes.delete, 54cad9e), which frees its EK binding and the machine's hardware key; the old node's journal.db is already aside (b4fcc2e), so bcd node does not refuse it as another node's. See recovering a node whose TPM changed. Nodes whose certificate names no AK (every other join) renew as before; a key of theirs in a cleared TPM is lost the same way and needs a new join (bcd join --reset --yes with a token).

Refused renewals are visible to the node owner since 0e42018: once the node proved its current key, JoinService publishes security.renew.refused naming the node, with the reason (for example renew:key-attestation-missing); see refused renewals. A renewal that fails on the node before any request (the TPM unavailable, another AK) reaches no server and publishes nothing; it shows only as renewal failed in bcd's log.

Certificate revocation

A revoked serial reaches every gateway within about a second through the revocation feed, which polls node_certs.revoked_at. The gateway sends the session RevocationNotice{reason, serials}, closes the node's tunnels, and ends the stream UNAUTHENTICATED. The daemon stops using the certificate and does not redial with it. New opens with the serial are refused at the handshake, and the Hello transaction checks node_certs directly, also refusing a certificate from an older identity generation of the node. PLAN.md's target is a revoked node out within 5 seconds.

Reconnect storm

When a gateway dies or a network heals, thousands of daemons redial at once. Four things keep that bounded: full-jitter back-off on the daemon, the handshake token bucket (200 per second per gateway), the pending-Hello bound (64 transactions in flight, beyond which a Hello is refused RESOURCE_EXHAUSTED and retried), and a small Hello that carries inventory hashes instead of inventory. Without the pending-Hello bound, a storm once queued enough Hello transactions to starve the gateway's own lease renewal until it fenced itself. Drain spreads planned reconnects over up to 60 seconds.

Compromised gateway

A gateway holds no CA and no signing key. A compromised one can delay, drop or read frames. It cannot:

• mint certificates: Renew needs a proof of possession by the node's current key, verified by JoinService;
• harvest join tokens: they are sealed to the JoinService encryption key, and JoinResponse is signed;
• read secrets: they travel as JWE to a one-time key of the node;
• forge attested results: they carry node_sig, verified against node_certs by workers;
• forge a GPU binding: with attestedBindings, a reservation goes active only on the node's own signed AllocationBound; an unsigned, foreign-signed or replayed report is rejected (reservation.binding-rejected), and a forged re-report of an active binding marks it lost instead of replacing it;
• forge intent: commands and NodeSpecs are signed by worker keys the daemon verifies; the gateway relays the stored signed spec bytes verbatim;
• open a tunnel of its own: the daemon verifies every exec ticket again against the JWKS pinned in its NodeSpec and its own jti set (bcd keeps it in its journal), and requires the ticket's sid to be the tunnel it is asked to open.

Signed commands and specs and attested bindings all come with pki.commandSigning on the server. Daemons require signatures for destructive kinds from v0.2 and for all kinds from v0.4, and for specs that change access from v0.2 and for every spec from v0.4. tests/e2e/attested-bindings.test.ts runs a compromised gateway's forged reports through a real gateway. See security.

Cloned identity

Two machines presenting the same certificate, a copied key, keep taking the node's session from each other: each Hello from the other boot wins and publishes security.node.duplicate-session. More than 3 such alternations within 10 minutes quarantine the certificate: it is revoked with reason duplicate-identity, both clones lose their sessions through the revocation feed, and security.node.quarantined is published. Workloads on the real machine keep running (nothing is released, I5); an operator re-joins it with a new token. See duplicate identity.

How the failures are exercised

better-compute sim --nodes 100 --chaos heavy --seed 7

The real Go daemon is tested at a smaller scale. Its unit tests cover the journal rules, crash recovery, the server time mark and the session handling against scripted streams, and tests/e2e/bcd.test.ts runs the fakehw build against a real gateway through a join, commands, a GPU binding and a reboot, plus five simulated machines joining through bcd sim (see testing). It does not run under chaos yet: bcd sim exists (fakehw builds, many real agents in one process), but PLAN.md's protocol and skew suite over lossy links and the nightly bcd sim --nodes 10000 against real gateways and PostgreSQL through Toxiproxy (v0.4) are not built.