# Failure modes (/docs/daemon/failure-modes) > What happens when a gateway, a node, the network, the journal, PostgreSQL, the api or a worker fails, or clocks and certificates go wrong, and which invariants hold. The daemon protocol is built on the assumption that everything fails: processes crash, links go half-open, databases fail over, clocks drift, journals get wiped. The design goal is that each failure costs a reconnect and a replay, never a duplicated side effect, a double booking, or a released allocation. This page lists what happens for each failure, what state is lost, and which invariant the behavior protects. ## The invariants [#the-invariants] PLAN.md ยง15 defines the fleet invariants. Six of them are checked mechanically after every simulated run, and a seventh after a simulated night, by pure functions in `@better-compute/testing` (`checkI1`, `checkI2`, `checkI3`, `checkI4`, `checkI5`, `checkI7`, `checkI8`): | Id | Invariant | Checked over | | -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | I1 | No overlapping bookings or node-reported assignments: no atom booked by two allocations at one instant, no device bound by a node for two allocations at once, and no binding of a device its allocation did not book | `device_bookings` rows and the bindings nodes reported | | I2 | At most one execution per command: a non-retry-safe or self-terminating command starts its side effect at most once; a retry-safe one re-executes only after a crash interrupted it; after a journal loss, a non-retry-safe command accepted in the lost journal never starts again and never ends `succeeded` | the side-effect facts the daemons record, journal losses, and outcomes | | I3 | Convergence within a bound: a delivered spec is reported observed within 60 s while the node stays reachable | specs delivered and status reports, from the simulator's links | | I4 | No false liveness positives: no node is marked `unreachable` or `lost` within 20 s of a gateway hearing from it, whatever the gateway's clock says | connectivity transitions against the frames the transport delivered | | I5 | No allocation released only because a node disconnected | release reasons against connectivity transitions | | I7 | Nothing pending that an offline solver could place (checked after the nightly simulation's quiet period) | capacity's `placeablePending` | | I8 | Billing conservation: the sum of usage equals the sum of booking duration times atoms, with no duplicate keys | allocations and flushed usage windows | I6 (latency) is part of the design but has no checker in this module; the real-time benchmarks measure it. ## At a glance [#at-a-glance] | Failure | What happens | Lost | Invariant kept | | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------- | | [Gateway crash](#gateway-crash) | Daemons redial elsewhere with jitter; `Hello` bumps the epoch and replays; 120 s grace before `unreachable` | Open streams and tunnels | I2, I5 | | [Gateway cut off from PostgreSQL](#gateway-cut-off-from-postgresql) | The gateway fences itself within 10 s and drops every stream; its stale writes fail on the epoch | Open streams | One router per node | | [Daemon crash](#daemon-crash-or-node-reboot) | systemd restarts it; the journal settles interrupted commands | Work in progress of non-retry-safe commands | I2 | | [Node reboot](#daemon-crash-or-node-reboot) | New `boot_id`: self-terminating commands complete on rejoin | Nothing | I2 | | [Network partition](#network-partition-and-half-open-links) | Session closes after three missed heartbeats; node `disconnected`, later `unreachable`; workloads keep running | Nothing | I5 | | [Journal loss](#journal-loss) | New `journal_id`: in-flight non-retry-safe commands fail `journal-lost` | Unacked events of the lost journal | I2 | | [PostgreSQL unavailable](#postgresql-unavailable) | Gateways fence and drop streams; daemons back off and keep enforcing their last spec; 5 min grace afterwards | Nothing durable | I5 | | [api or worker crash](#api-or-worker-crash) | Stateless api restarts; worker leases expire and move | Nothing durable | none needed | | [Clock skew](#clock-skew) | Ages instead of timestamps; server time on the daemon; `ClockSkew` on control-plane processes | Nothing | I4 | | [Certificate expiry](#certificate-expiry) | Renewal at 60% of the lifetime; the session closes at expiry | The session | | | [TPM cleared or unavailable](#tpm-cleared-or-unavailable) | A certificate naming the TPM's AK does not renew without that AK; bcd retries and fails loudly, an operator re-joins it with `bcd join --reset --yes --attestor tpm` | The session, until the re-join | | | [Certificate revocation](#certificate-revocation) | `RevocationNotice`, then the stream closes; the daemon stops redialing | The session | | | [Reconnect storm](#reconnect-storm) | Token bucket, pending-Hello bound, jitter, hash-only `Hello` | Nothing | Bounded load | | [Compromised gateway](#compromised-gateway) | Can delay, drop or read frames; cannot mint certificates, read tokens or secrets, or forge results, bindings or signed intent | | P14 | | [Cloned identity](#cloned-identity) | The sessions alternate; more than 3 alternations in 10 minutes revoke the certificate | Both clones' sessions | I5 | ## Gateway crash [#gateway-crash] A gateway process dies: no disconnect writes, no lease release. Everything it held in memory is gone, and it held nothing else. 1. Each daemon sees its stream end and redials with full-jitter back-off (0 to 1 s, doubling to 60 s). The L4 load balancer sends it to a live gateway. 2. The new gateway's `Hello` transaction bumps the epoch and routes the node to itself. In-flight commands (`dispatched`, `acked`, `running`) move to the new epoch and are replayed in `seq` order, except those the daemon reports as running; its unacked results are applied first. Pending commands need no rewrite: dispatch is a join on `node_sessions.gateway_id`, which now names the new gateway. 3. The daemon's journal deduplicates every replay: a done command resends its result, a running one re-attaches. Nothing runs twice (I2). 4. Nodes that do not come back are handled by the Liveness controller: once the dead gateway's `gateway:` lease expires (20 s), its nodes become `disconnected`, and only after the 40-second lease TTL plus a 120-second gateway-loss grace do they become `unreachable`. That transition adds a NoSchedule reason and a condition and releases nothing (I5). Tunnels on the dead gateway close; the caller opens a new one. In the fleet simulator with three real gateways and the real control plane, killing one re-homed its nodes with a p99 of 1 second at 1,000 nodes and 13.25 seconds at 10,000, with no invariant violations (`bench/multi-gateway`). PLAN.md's v0.4 target is under 60 seconds with no unreachable marks and no duplicate executions. ## Gateway cut off from PostgreSQL [#gateway-cut-off-from-postgresql] A gateway that loses the database, or stalls, cannot renew its `gateway:` lease. After half the lease TTL (10 seconds) without a successful renewal, or as soon as it finds another owner on the lease, it fences itself: it forgets its lease and closes every stream `UNAVAILABLE`. Its daemons redial elsewhere. A gateway that does not know it is stale (a paused process that wakes up) is stopped by two fences: * every routing transaction begins by re-reading its lease `FOR SHARE`; a lease that moved aborts the transaction (`FencedError`), which fences the gateway; * routing and liveness writes carry `AND session_epoch = $myEpoch`. After the node's `Hello` elsewhere, the zombie's dispatch marks and disconnect writes match nothing, and its liveness flush reports those nodes as gone, so it closes their streams `ABORTED`. Dispatch holds the session row `FOR SHARE`, so it cannot race an epoch bump. Results and events the zombie still ingests are facts and are accepted from any epoch, deduplicated as usual. At most one gateway routes to a node at a time. `tests/sim/multi-gateway.test.ts` runs a stalled gateway that comes back as a zombie holding sessions its nodes have left. ## Daemon crash or node reboot [#daemon-crash-or-node-reboot] **Daemon crash.** systemd restarts bcd after 5 seconds. The new process has a new incarnation id, and `journal.Recover` settles what the old one left behind: | Left in the journal | Outcome | | --------------------------------------- | ------------------------------------------------------------------------------------- | | `done` | Its result is replayed in `Hello.unacked_results` until acked | | `accepted` or `running`, retry-safe | Waits for the server to redeliver it, then runs again after the acceptance checks | | `accepted` or `running`, not retry-safe | `FAILED` with `interrupted`; the owning controller decides | | `running`, self-terminating, same boot | Treated like any other entry by its retry-safety (the process died before the reboot) | The rule that makes this safe: `accepted` is fsynced before any side effect starts. A command is either known to the journal before it could have done anything, or it never ran. **Node reboot.** The kernel `boot_id` changes. A self-terminating command (`node.reboot`, `boot.kexec`, `daemon.upgrade`) journaled `running{boot_id}` immediately before its side effect, so on restart the daemon reports it `SUCCEEDED` with `rebooted`, and the server's `Hello` transaction completes any in-flight self-terminating command the same way when it sees the new `boot_id` or role. Neither side ever replays it. Files under `/run` are gone after a reboot; the devices converger rewrites missing env files for assignments it still holds. **Self-update.** A `daemon.upgrade` restarts bcd without a reboot, so the boot id does not change; bcd decides it from `upgrade.json` instead (de9f380). The new binary reports `SUCCEEDED` once its first `Welcome` arrives. A binary that crashes or never gets a `Welcome` within 120 seconds is swapped back (under systemd a crash loop also triggers `bcd-rollback.service`), and the old binary reports `FAILED` with `update-failed` and never tries that version again. On a node under `bcd helper`, the helper's own watchdog also swaps back an update still unconfirmed 30 seconds after its deadline and restarts `bcd.service`, so a new binary that never gets as far as asking cannot keep the node on it (2d67b25). The restarted process decides by where the `current` link points, never by the version it claims to run, so a lying binary cannot mark the new version failed and disarm the watchdog (c139b9c). **A bad release that confirmed.** The swap back covers only an update that never confirmed. Once the new binary confirms, the release floor rises to its sequence and bcd refuses every older binary, the one still in its other slot included, so neither the watchdog nor a rollout can take the node back (c139b9c). `rollouts.abort` with `rollback` returns only nodes still upgrading (8347a3d). A release that confirmed and then misbehaves is fixed by rolling forward: sign and register a newer release with a higher sequence and roll it out; see [rollouts](/docs/guides/rollouts#pause-resume-abort). See [self-update](/docs/daemon/install#self-update). **The helper is down or refuses.** A `bcd node` running unprivileged beside [`bcd helper`](/docs/daemon#privilege-separation) (28dc2ae) keeps its session and journal when the helper is gone, but every operation it sends there (a CDI spec or env file, the sshd files, the nftables table, a GPU reset, a reboot, a self-update step) fails with an error that starts `helper:`, like the direct operation failing. Such an error is not terminal, so a `devices` or `access` section stays `PROGRESSING` and is retried on the next pass (the spec is not observed meanwhile), and a command such as `gpu.reset` or `node.reboot` fails as it would without root. A node and a helper of different protocol versions refuse each other's messages, so restart both after replacing the binary. On a node installed with `--unprivileged`, `bcd-helper.service` restarts the helper after 2 seconds (`Restart=always`), and `bcd.service` `Requires=` it, so stopping the helper stops the node too; see [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged). ## Network partition and half-open links [#network-partition-and-half-open-links] When a node's link is cut, both ends notice through silence. The gateway closes the session after three missed heartbeats (30 seconds) and writes an epoch-fenced disconnect; HTTP/2 PINGs every 20 seconds detect a dead TCP connection. A half-open connection, where one end died silently, ends the same way, and when the node comes back its new session supersedes the old one by epoch. While partitioned, the daemon keeps enforcing its last desired state and keeps every device assignment: it never drops one on its own. Workloads keep running. It journals events (up to 64 MiB and 10,000 unacked; metric events are dropped oldest-first when full, results and transitions never) and redials with back-off. On the control plane the node goes `disconnected`, then `unreachable` after the 40-second lease TTL, with a NoSchedule reason and the `NodeUnreachable` condition. Only nodes of `provider` and `vps-host` pools can become `lost`, and only when the pool sets `policy.lostAfter`; metal never does automatically. No step releases an allocation, a booking or a command (I5). Commands that pass `not_after` while the node is away are swept to `expired`, or rejected `EXPIRED` by the daemon if they were already delivered. ## Journal loss [#journal-loss] A ramdisk reboot, a reinstall or corruption loses the journal. The daemon starts a new one with a new `journal_id`; its identity stays with its key, so it reconnects as the same node. bcd moves a journal file that does not open aside (`journal.db.corrupt-`) rather than deleting it, and starts its server time mark from the copy in `identity.json`, so a lost journal does not reopen the clock-rewind window. * The server sees the new `journal_id` in `Hello` and fails every in-flight command that is not retry-safe with reason `journal-lost`, instead of replaying it into a journal that cannot deduplicate. Controllers re-decide (I2). * Retry-safe in-flight commands are replayed and may run again, which their kind allows. * The event ack restarts at 0 for the new journal. Events still unacked in the lost journal are gone. * The new journal holds no spec, so the daemon reports applied generation 0 and receives a full `DesiredState` at `Hello`. The fleet simulator's `journalLoss` chaos and `sim.loseJournal(nodeId)` exercise this path. ## PostgreSQL unavailable [#postgresql-unavailable] PostgreSQL is the only source of truth, so an outage stops the fleet from changing without losing anything. * **Gateways** cannot renew their leases and fence themselves within 10 seconds, dropping every stream. Without a lease they refuse new sessions `UNAVAILABLE`. A frame that fails to write also drops its stream. The lease loop keeps retrying, and once it renews, the gateway admits sessions again. * **Daemons** back off up to 60 seconds between attempts, keep enforcing their last spec, and keep results and events in the journal until a gateway acknowledges them. Nothing is buffered in the gateway beyond its memory. * **Commands** stay `pending`; nothing is dispatched. After the outage, those past `not_after` are swept to `expired`, and a stale command that does reach a node is rejected on `not_after` or `min_generation`. * **Liveness.** When the Liveness controller finds a gap of more than 20 seconds since its last sweep, it holds every timed transition back for 5 minutes, so the ages the gateways could not flush do not read as dead nodes. PLAN.md sizes this for a database failover of about 30 seconds: the fleet reconnects, nothing is marked unreachable, and nothing is released. ## api or worker crash [#api-or-worker-crash] **api.** api processes are stateless; a crash fails the requests in flight, and clients retry with their idempotency keys. `Join`, `Renew` and secret requests reach JoinService and the secret broker through the gateway's `GatewayServices` port. While the process serving them is down, `Join` and `Renew` fail and the daemon retries (a node certificate lasts 24 hours and renews at about 60% of its life, which leaves hours of margin), and secret requests are answered with the error `unavailable`. **Worker.** Workers run the reconcilers, the CommandSweeper and the Liveness controller under table leases (15 seconds for shards, renewed every 5). When a worker dies, its leases expire and another worker takes them over; the work queue loses no wake-up because a claim that is not completed simply becomes claimable again. While no worker runs, gateways keep dispatching commands that are already pending and keep ingesting results, but nothing new is decided and nothing is swept. The Liveness controller's outage grace (above) applies when it comes back. See [leases and fencing](/docs/guides/concepts/consistency#leases-and-fencing). ## Clock skew [#clock-skew] No component trusts another's wall clock. * **Gateways report ages.** Liveness is flushed as "seen `age_ms` ago", and PostgreSQL computes `compute_now() - age`. A gateway whose clock is off cannot make a node look alive or dead (I4). * **Daemons use server time.** `Welcome.server_time` gives the daemon the offset between its clock and `compute_now()`, and it evaluates `not_after` through that offset. Conditions carry `age_ms` on the daemon's monotonic clock, and the control plane derives `since` itself; event and inventory timestamps are advisory. Because `server_time` is unsigned, the daemon keeps a server time mark that only moves forward and refuses a `Welcome` more than 5 minutes before it, so a gateway cannot rewind the daemon into an expired signer's window (see [Welcome](/docs/daemon/sessions#welcome)). * **Control-plane processes** anchor their business clock to `compute_now()`, re-sampled at every lease renewal. A process more than 1 second off raises `ClockSkew`: a log warning and a `selfCheck` finding, and the `clock_offset_seconds` metric shows the offset. * **Proofs** tolerate 60 seconds: `Renew` proofs of possession, WebSocket proofs. ## Certificate expiry [#certificate-expiry] Node certificates last at most 24 hours (ramdisk certificates at most 2), and bcd renews at about 60% of the lifetime with `Renew`, proving possession of its current key. A session whose certificate expires is closed `UNAUTHENTICATED` ("certificate expired; renew and reconnect") by the gateway's liveness loop. A certificate that expired while the machine was offline cannot be renewed. From v0.2, when the server has the `x509pop` attestor configured, the machine re-joins with its old key within 7 days unless the certificate was revoked. See [renewal](/docs/guides/nodes/joining#renewal). Gateway serving certificates last 30 days in the PLAN.md design, which lists serving-certificate expiry among the fleet checks of `doctor`. ## TPM cleared or unavailable [#tpm-cleared-or-unavailable] A node that joined with the `tpm` attestor and a certified node key has a certificate naming its TPM's attestation key (`urn:bc:tpm-ak:`), and since d94f1f3 it renews only with a next key that AK certifies. bcd never falls back to a file key for such a certificate: * **The TPM cannot be reached** (a missing `/dev/tpmrm0`, or, for an unprivileged `bcd node` under `[helper]`, a `bcd helper` that is down or cannot open the TPM: since 8870b8d that node reaches the TPM only through the helper). The renewal fails before any request: "the certificate's node key is in the TPM, which is unavailable" when bcd opens the TPM itself, or `renewal:` with the helper's error. bcd logs `renewal failed` with the attempt and `expiresIn`, and retries with backoff from 30 seconds to 10 minutes. Once the TPM answers again, the next retry renews. A TPM-held current key needs the TPM to load too, so a restarted `bcd node` does not start without it. * **The TPM derives another AK** (another TPM, or one whose endorsement seed changed). The renewal fails the same way, with "this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join" (a1c1a46), and retrying cannot help. * **The TPM was cleared** (`TPM2_Clear`). A clear replaces the owner hierarchy's seed but not the endorsement one, so the AK survives and the node key does not: the key is a primary of the owner hierarchy, the stored record derives another key, and bcd refuses its identity with "the node key does not match the node certificate: the TPM derives another key from node.key (was the TPM cleared?); join again: bcd join --reset --yes, then bcd join". Since a1c1a46 the cleared-TPM hint is here, where a clear shows, rather than on the AK check. `bcd doctor` fails its `identity` check with the same text. Either way the certificate runs out within its 24 hours and the session closes as in [certificate expiry](#certificate-expiry). The recovery is to join again with the TPM, as root: `bcd join --reset --yes --attestor tpm` moves the old identity and journal aside (renamed `.reset-`, not deleted) and joins, and the attestor reclaims the node bound to the machine's EK, with a new identity generation, a new certified key and a fresh AK statement. A machine whose EK changed (another TPM) joins as a new node instead, once the old node is cordoned and deleted (`better-compute nodes delete --yes`, 4049621; `nodes.delete`, 54cad9e), which frees its EK binding and the machine's hardware key; the old node's `journal.db` is already aside (b4fcc2e), so `bcd node` does not refuse it as another node's. See [recovering a node whose TPM changed](/docs/guides/nodes/attestation#recovering-a-node-whose-tpm-changed). Nodes whose certificate names no AK (every other join) renew as before; a key of theirs in a cleared TPM is lost the same way and needs a new join (`bcd join --reset --yes` with a token). Refused renewals are visible to the node owner since 0e42018: once the node proved its current key, JoinService publishes `security.renew.refused` naming the node, with the reason (for example `renew:key-attestation-missing`); see [refused renewals](/docs/operations/security#refused-renewals). A renewal that fails on the node before any request (the TPM unavailable, another AK) reaches no server and publishes nothing; it shows only as `renewal failed` in bcd's log. ## Certificate revocation [#certificate-revocation] A revoked serial reaches every gateway within about a second through the revocation feed, which polls `node_certs.revoked_at`. The gateway sends the session `RevocationNotice{reason, serials}`, closes the node's tunnels, and ends the stream `UNAUTHENTICATED`. The daemon stops using the certificate and does not redial with it. New opens with the serial are refused at the handshake, and the `Hello` transaction checks `node_certs` directly, also refusing a certificate from an older identity generation of the node. PLAN.md's target is a revoked node out within 5 seconds. ## Reconnect storm [#reconnect-storm] When a gateway dies or a network heals, thousands of daemons redial at once. Four things keep that bounded: full-jitter back-off on the daemon, the handshake token bucket (200 per second per gateway), the pending-Hello bound (64 transactions in flight, beyond which a `Hello` is refused `RESOURCE_EXHAUSTED` and retried), and a small `Hello` that carries inventory hashes instead of inventory. Without the pending-Hello bound, a storm once queued enough `Hello` transactions to starve the gateway's own lease renewal until it fenced itself. `Drain` spreads planned reconnects over up to 60 seconds. ## Compromised gateway [#compromised-gateway] A gateway holds no CA and no signing key. A compromised one can delay, drop or read frames. It cannot: * mint certificates: `Renew` needs a proof of possession by the node's current key, verified by JoinService; * harvest join tokens: they are sealed to the JoinService encryption key, and `JoinResponse` is signed; * read secrets: they travel as JWE to a one-time key of the node; * forge attested results: they carry `node_sig`, verified against `node_certs` by workers; * forge a GPU binding: with `attestedBindings`, a reservation goes active only on the node's own signed `AllocationBound`; an unsigned, foreign-signed or replayed report is rejected (`reservation.binding-rejected`), and a forged re-report of an active binding marks it lost instead of replacing it; * forge intent: commands and NodeSpecs are signed by worker keys the daemon verifies; the gateway relays the stored signed spec bytes verbatim; * open a tunnel of its own: the daemon verifies every exec ticket again against the JWKS pinned in its NodeSpec and its own `jti` set (bcd keeps it in its journal), and requires the ticket's `sid` to be the tunnel it is asked to open. Signed commands and specs and attested bindings all come with `pki.commandSigning` on the server. Daemons require signatures for destructive kinds from v0.2 and for all kinds from v0.4, and for specs that change `access` from v0.2 and for every spec from v0.4. `tests/e2e/attested-bindings.test.ts` runs a compromised gateway's forged reports through a real gateway. See [security](/docs/operations/security). ## Cloned identity [#cloned-identity] Two machines presenting the same certificate, a copied key, keep taking the node's session from each other: each `Hello` from the other boot wins and publishes `security.node.duplicate-session`. More than 3 such alternations within 10 minutes quarantine the certificate: it is revoked with reason `duplicate-identity`, both clones lose their sessions through the revocation feed, and `security.node.quarantined` is published. Workloads on the real machine keep running (nothing is released, I5); an operator re-joins it with a new token. See [duplicate identity](/docs/daemon/sessions#duplicate-identity). ## How the failures are exercised [#how-the-failures-are-exercised] - [createFleetSim](/docs/operations/testing): N FakeDaemons on seeded fake hardware and in-memory links with drops, duplicates and reordering, against a gateway stub or a real gateway. Chaos: crashes, reboots, journal loss, partitions, isolation, half-open links, resets, stalls and gateway kills. Same seed, same event log. - [createControlPlaneSim](/docs/operations/testing): The same simulator with the real control plane on the sim clock and real gateways; facts for the checkers come from compute's own tables. - [better-compute sim](/docs/reference/cli): Runs the fleet simulator with a steady workload and `--chaos none`, `light` or `heavy`, then checks the simulator's invariants (I1, I2, I3, I4, I5 and I8) and prints the event-log digest. - [The nightly and bench suites](/docs/operations/testing): `tests/sim/multi-gateway.test.ts` (a kill mid-command, a zombie gateway, a revocation), the nightly simulation with worker outages, and `bench/multi-gateway` at up to 10,000 nodes. ```bash better-compute sim --nodes 100 --chaos heavy --seed 7 ``` The real Go daemon is tested at a smaller scale. Its unit tests cover the journal rules, crash recovery, the server time mark and the session handling against scripted streams, and `tests/e2e/bcd.test.ts` runs the fakehw build against a real gateway through a join, commands, a GPU binding and a reboot, plus five simulated machines joining through [`bcd sim`](/docs/daemon#bcd-sim) (see [testing](/docs/operations/testing#the-go-daemon)). It does not run under chaos yet: `bcd sim` exists (fakehw builds, many real agents in one process), but PLAN.md's protocol and skew suite over lossy links and the nightly `bcd sim --nodes 10000` against real gateways and PostgreSQL through Toxiproxy (v0.4) are not built.