# PKI (/docs/operations/pki) > The local CAs, JoinService keys and signing keys a deployment owns, the pki CLI and bundle, SPIFFE IDs, node certificate lifetimes, revocation and rotation. Better IAM identifies people and AI agents. Machines are identified by Better Compute's own PKI: a set of local certificate authorities and keys that one deployment owns, created once with `better-compute pki init` and loaded by every process at start. Everything is ECDSA P-256 with SHA-256, except the two SSH CAs, which are Ed25519 by default, and the package needs nothing beyond `node:crypto`. The design rule is that **no key does two jobs**. A CA that signs agent certificates cannot sign a gateway's server certificate, and the key that signs `JoinResponse` is not the one join tokens are sealed to. ## The keys [#the-keys] | Key | Signs or does | Lifetime | Who trusts it | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------- | | Root CA | Only the three subordinate CAs | 10 years | Anchor above the CAs; can be kept offline | | Node CA | Agent client certificates: one SPIFFE URI SAN (then only `urn:bc:` key statements), the role as `OU=bc-role:`, `clientAuth` | CA 1 year; leaves at most 24 h (ramdisk at most 2 h) | Gateways, as the trust anchor for daemon mTLS | | Gateway serving CA | Gateway server certificates: DNS SANs, `serverAuth` | CA 1 year; leaves at most 30 days | Daemons, by its SPKI pin | | Internal CA | Control-plane mutual TLS between api, workers and gateways | CA 1 year; leaves at most 7 days | The control plane | | JoinService signing key | `JoinResponse` and `SignedTrustBundle` | Certificate 10 years | Daemons, by pin | | JoinService encryption key | Nothing: join tokens are sealed to it | — | Daemons, by pin | | Command-signing root (v0.2) | 24 h worker keys, which sign every command | 5 years | Daemons, through the trust bundle | | Exec ticket key (v0.3) | 60 s single-use tunnel tickets | — | Gateways (their tunnel JWKS) and daemons (the JWKS in NodeSpec) | | SSH CA (v0.3) | Break-glass OpenSSH user certificates, at most 1 h | — | Nodes' sshd, through NodeSpec | | Sandbox SSH CA (v0.5) | `sandboxes.exec` certificates, at most 15 minutes, for `@` | — | Sandbox guests' sshd, through the user data vps renders | | Audit signing key (v0.4) | Per-tenant Merkle checkpoints of `audit_events` | Certificate 10 years | `audit verify` and third parties, by pin | Subordinate CAs never outlive the root: a CA certificate's `notAfter` is capped at the root's. Leaf lifetimes are enforced by the CA itself, whatever a caller asks for. The gateway holds none of these keys in its own code: it relays `Join`, `Renew` and secret requests to JoinService and never signs anything. See [security](/docs/operations/security#gateway-processes) for what that means for the process that runs it. ## The pki commands [#the-pki-commands] ### pki init [#pki-init] ```bash export BETTER_COMPUTE_SECRET=$(openssl rand -base64 48) # the same for every process better-compute pki init --trust-domain compute.example.com ``` It creates every key above and writes one JSON file, mode 0600, never overwriting an existing one. The private keys are sealed with AES-256-GCM under a key derived (HKDF-SHA256) from `BETTER_COMPUTE_SECRET`, with the trust domain in the authenticated data. The sealed payload also carries a SHA-256 over every public field (certificates, trust anchors, the trust version), so whoever can write the file but lacks the secret cannot add a command-signing root or pin a trust version: loading refuses a sealed bundle whose public parts changed (`INVALID_CONFIG`; restore it from a backup), while reformatting the JSON is harmless. Any CLI write-back re-seals in the current format. `pki init` prints the pins, like `pki show`. > **Re-seal bundles sealed before 482dea8.** A bundle sealed in the former format (before the seal covered the public parts, commits 482dea8 and 7b9e9af) is refused wherever it is loaded (`INVALID_CONFIG`: "sealed in the former format"), and so is an unsealed bundle when `BETTER_COMPUTE_SECRET` is set, since a sealed one may have been replaced. Nothing authenticates such a file: whoever could write it could have added a trust anchor or a second SSH CA line. Once, with `BETTER_COMPUTE_SECRET` set: 1. run [`pki reseal`](#pki-reseal). For such a bundle it prints the pins it would vouch for, with `authenticated: false`, and stops with `INVALID_STATE`; 2. compare those pins with your records (install scripts, join tokens, pinned CAs); 3. run `pki reseal --accept-unauthenticated`, which writes the bundle in the current format. `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1` only lets other readers, such as a server, load the legacy bundle until it is resealed; never leave it set on a running deployment. The commands that write the bundle back (`pki rotate-command-root`, `pki add-sandbox-ca`, `pki renew-ca`) refuse it with `INVALID_STATE`: only `pki reseal` vouches for an unauthenticated bundle. A former-format bundle that carries command-signing rotation fields cannot be re-sealed: restore the copy from before the rotation and run `pki rotate-command-root` again. | Flag | Default | Meaning | | ----------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------ | | `--trust-domain DOMAIN` | required (`BETTER_COMPUTE_TRUST_DOMAIN`) | The SPIFFE trust domain of agent identities | | `--pki PATH` | `.better-compute/pki.json` (`BETTER_COMPUTE_PKI`) | Where to write the bundle | | `--offline-root PATH` | none | Write the root key to its own file (sealed like the bundle) and leave it out of the bundle | | `--organization NAME` | `better-compute` | Subject organization of the CA certificates | | `--plaintext` | off | Store the private keys as PEM when `BETTER_COMPUTE_SECRET` is not set | Without the secret and without `--plaintext` it refuses with `MISSING_ENV`. Keep the `--offline-root` file off the servers: it is needed only to replace a CA. ### pki show [#pki-show] `pki show` reads the public half of the bundle. With `BETTER_COMPUTE_SECRET` it first checks that half against the seal and prints `authenticated: true`; without the secret it still prints the pins, with `authenticated: false` and a warning that whoever can write the file chooses them. `audit verify` and `daemon install-script` read pins the same way. It prints: | Field | Used by | | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `trustDomain` | Install scripts, `bcd` configuration | | `servingCaSpkiPin` | Daemons check the gateway's certificate against it | | `joinSigningKeyPin`, `joinEncryptionKeyPin` | Daemons check `JoinResponse` and seal join tokens only to a matching key | | `joinEncryptionJwk` | The key daemons seal join tokens to | | `auditSigningKeyPin` | `audit verify` | | `execTicketKeyId` | The `kid` of the exec ticket key (its RFC 7638 thumbprint) | | `sshCaPublicKey` | The OpenSSH public key sshd trusts | | `sandboxSshCaPublicKey` | The sandbox SSH CA's public key, which sandbox guests trust (`vps({ sandboxExec: { caPublicKey } })`) | | `commandSigningRootSha256` | The command-signing root, as trust bundles list it | | `commandSigningNextRootSha256`, `commandSigningPreviousRootSha256`, `trustVersion` | During a root rotation: the root published but not signing yet, the former root still trusted, and the trust bundle's version | | `expires` | When the root, node, serving, internal and command-signing certificates expire | | `certificates` | The hex SHA-256 of every certificate (DER) of every stored chain, in file order. Pins name only a chain's first certificate, so this is what shows a certificate appended behind it | Pins have `bcd`'s form, `sha256/`. `better-compute daemon install-script` reads the bundle and writes the serving CA pin and the JoinService pins and key into each machine's configuration. ### pki issue-gateway [#pki-issue-gateway] ```bash better-compute pki issue-gateway --san gw.eu-1.compute.example.com --out /etc/better-compute/tls --days 14 ``` It generates a P-256 key and issues a serving certificate from the bundle's serving CA. It writes `NAME.key` (mode 0600\) and `NAME.crt` (the certificate followed by the serving CA, stopping below the root) into `--out`, replacing earlier files atomically. | Flag | Default | Meaning | | ------------- | -------------------------- | ------------------------------------- | | `--san NAMES` | required | DNS names, comma-separated | | `--out DIR` | `.better-compute/gateway` | Directory for the key and certificate | | `--name NAME` | `gateway` | File name stem | | `--days N` | 30 | Lifetime, from 1 to 30 days | | `--pki PATH` | `.better-compute/pki.json` | The bundle | ### pki add-sandbox-ca [#pki-add-sandbox-ca] ```bash better-compute pki add-sandbox-ca ``` Bundles from `pki init` already hold a [sandbox SSH CA](#the-sandbox-ssh-ca). For a bundle made before sandboxes, this creates one (Ed25519, comment ` sandbox ssh ca`), writes the bundle back (sealed when `BETTER_COMPUTE_SECRET` is set; an offline root stays out) and prints `sandboxSshCaPublicKey`. A bundle that already has one is `INVALID_STATE`. It writes the bundle back, so it refuses `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1` (`INVALID_STATE`): re-seal a former-format bundle with [`pki reseal`](#pki-reseal) first. Roll the bundle out to every api replica, and give vps the public key as `sandboxExec.caPublicKey`. The only flag is `--pki`. ### pki reseal [#pki-reseal] ```bash better-compute pki reseal # an unauthenticated bundle: prints its pins and stops better-compute pki reseal --accept-unauthenticated # after comparing them with your records ``` It rewrites the bundle sealed with `BETTER_COMPUTE_SECRET` (required, at least 32 characters; `MISSING_ENV` otherwise) in the current format, where the seal also covers every certificate and pin, so a file changed without the secret no longer loads. It keeps every key and certificate: nothing is rotated. * **An authenticated bundle** (already sealed in the current format under this secret) is simply resealed. * **An unauthenticated bundle** (sealed in the former format, or unsealed while `BETTER_COMPUTE_SECRET` is set) is not sealed at first: resealing would vouch for whatever the file holds. The command prints the pins it would vouch for, the same fields as `pki show` with `authenticated: false`, and stops with `INVALID_STATE` ("nothing vouches for the pins above"). Compare them, and the `certificates` digests (which also reveal a certificate appended behind a pinned one), with your install scripts, join tokens and pinned CAs, then run it again with `--accept-unauthenticated`. After sealing it prints the pins it now vouches for, with `authenticated: true`. It needs no `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED`. | Flag | Default | Meaning | | -------------------------- | -------------------------- | -------------------------------------------------------------------------------- | | `--accept-unauthenticated` | off | Seal a bundle nothing authenticates, after comparing the pins printed without it | | `--pki PATH` | `.better-compute/pki.json` | The bundle | ### pki renew-ca [#pki-renew-ca] ```bash better-compute pki renew-ca --ca node better-compute pki renew-ca --ca serving --root-key /media/usb/root-ca.key ``` It issues a new certificate for the node, serving or internal CA from the root, for the same key and subject, and writes the bundle back (sealed when `BETTER_COMPUTE_SECRET` is set). Certificates the CA already issued stay valid and the serving CA's SPKI pin does not change, so nothing is re-pinned and no node rejoins. Roll the bundle out to every process before the old certificate expires. It prints the CA, the new `notAfter` and serial, and for `serving` the unchanged pin. Like every command that writes the bundle back, it refuses `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1` with `INVALID_STATE`. | Flag | Default | Meaning | | ----------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------- | | `--ca CA` | required | `node`, `serving` or `internal` | | `--root-key PATH` | none | The offline root key (`pki init --offline-root`), when the bundle has none; it is not written into the bundle | | `--days N` | 365 | Lifetime of the new certificate, 1 to 3650 days, capped by the root | | `--pki PATH` | `.better-compute/pki.json` | The bundle | ### pki trust-version [#pki-trust-version] ```bash better-compute pki trust-version ``` It reads the trust bundle the deployment last published (`trust_bundle_state`; its version [follows its content](#signed-intent)) through the deployment's configuration and compares it with this configuration and PKI bundle (03bdbc9). It prints `version` (the published one), `configurationMatches` (whether this bundle's roots and the configured quorum publish the same bundle), `nextVersion` (what this configuration would publish: the same version, or the next one), `approverSetEpoch` (the published approver set's epoch), `approverSetSha256` (this configuration's set digest, for [pinning the first set](#pinning-the-first-set), with `publishedApproverSetSha256` while they differ), `approverSetBehind` and `approverSetSample` (connected nodes enforcing another set, as the gateway records it, a node without keys included; see [nodes that hold another set](#changing-the-approver-set); 6612496, 08eb62e), `quorumChanged` and, when it did, `signQuorumForEpoch` (the epoch to sign the change for with [`batch sign-approvers`](#changing-the-approver-set), one above the published one; c6214dc), and how many connected daemons there are and how many are behind that version. Before any process has published a bundle it is `NOT_FOUND`. `--pki PATH` names the bundle. ### Reloading gateway certificates [#reloading-gateway-certificates] A running gateway picks a reissued certificate up without a restart when it reads it from files. Point `host.gateway.tlsFiles` at them (and `host.gateway.internal.tlsFiles` at the internal-CA pair of the control API): ```ts title="better-compute.config.ts" export const host = defineHost(() => ({ gateway: { region: 'eu-1', tls: {}, // other listener options; key, cert and ca come from the files tlsFiles: { key: '/etc/better-compute/tls/gateway.key', cert: '/etc/better-compute/tls/gateway.crt', ca: '/etc/better-compute/tls/node-ca.pem', // node CA roots, for client verification intervalMs: 60_000, // the default }, }, })); ``` `serve` reads the files at start over `tls` and then checks their modification time and size every `intervalMs`. A changed set is loaded into a new secure context first and swapped onto the daemon listener and the WebSocket fallback (or the control API): new handshakes get the renewed certificate, and open sessions keep theirs, so no daemon is disconnected by a rotation. A half-written or mismatched pair is reported (`Gateway serving certificate not reloaded: …`) and the listener keeps its certificate; the next change is tried again. Without `tlsFiles`, the certificate in `tls` is read once at start, so after reissuing, restart gateways one at a time. Outside `serve`, `serveGateway`, `serveGatewayWs` and `serveInternal` from `better-compute/gateway` return `setTls({ key, cert, ca? })`, which applies new material over the listener's original TLS options (protocol versions and the rest stay) and throws `INVALID_CONFIG`, changing nothing, when the key does not match its certificate. `watchTlsFiles({ key, cert, ca? }, apply, { intervalMs?, onError?, onReload? })` polls the files and calls `apply` with a changed set, for example `(tls) => served.setTls(tls)`. ## The bundle in code [#the-bundle-in-code] The bundle is a JSON document (`format: "better-compute-pki"`, `version: 1`) with the certificates and public keys in the clear and the private keys either sealed or, with `--plaintext`, as PEM. | Function | Package | Does | | --------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `loadPkiBundle(path, env)` | `better-compute/cli` | Reads the file and unseals it with `env.BETTER_COMPUTE_SECRET` (and `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1`, which loads a legacy bundle until it is resealed) | | `importPkiBundle(text, { secret?, rootKey?, allowUnauthenticated? })` | `better-compute/pki` | Loads a bundle; `INVALID_CONFIG` on a wrong secret, a damaged file, a key that does not match its certificate, changed public parts, or (unless `allowUnauthenticated`) a former-format seal or an unsealed bundle with a secret | | `exportPkiBundle(bundle, { secret?, omitRootKey? })` | `better-compute/pki` | Serializes a bundle, sealed when `secret` is given | | `exportRootKey(bundle, { secret? })` | `better-compute/pki` | The root key alone, for keeping it offline | | `readPkiPins(text, { secret? })` | `better-compute/pki` | The pins and expiry dates from the public half, the SHA-256 of every stored certificate (`certificates`), and `authenticated` (checked against the seal when `secret` is given) | | `createPkiBundle({ trustDomain })` | `better-compute/pki` | What `pki init` runs | | `bootstrapPins(bundle)` | `better-compute/pki` | `{ servingCaPin, joinSigningPin, joinEncryptionPin }` in bcd's `sha256/` form, taken from the bundle: what a new machine pins before it trusts anything the control plane says (`bcd join --serving-ca-pin --join-signing-pin --join-encryption-pin`), never the pins in the unsigned `/.well-known/better-compute`. Pass it to `provisionModule({ userData: { pins } })`, as the examples do; see [joining from the well-known document](/docs/daemon/install#joining-from-the-well-known-document) | | `tunnelsOptions(bundle, { issuer })` | `better-compute/pki` | The `tunnels` options: the exec ticket issuer, `ssh` (break-glass) and `sandboxSsh`, when the bundle has them | Wire it into the options and the gateway: ```ts title="better-compute.config.ts" const bundle = await loadPkiBundle(env.BETTER_COMPUTE_PKI ?? '.better-compute/pki.json', env); export default defineConfig(() => ({ // … pki: { nodeCa: bundle.nodeCa, // the gateway role's trust anchor and revocation set join: pkiJoinCrypto({ trustDomain: bundle.trustDomain, nodeCa: bundle.nodeCa, signingKey: bundle.joinSigning, encryptionKey: bundle.joinEncryption.privateKey, }), commandSigning: bundle.commandSigning, // v0.2 signed intent }, // The ticket issuer must be baseURL's origin: the api accepts only tickets from it. // Equivalent to { issuer: execTicketIssuer(bundle, …), ssh: bundle.sshCa, sandboxSsh: bundle.sandboxSshCa }. tunnels: tunnelsOptions(bundle, { issuer: 'https://compute.example.com' }), modules: [ auditModule({ signer: bundle.auditSigning }), vps({ images, sandboxExec: { caPublicKey: bundle.sandboxSshCa!.publicKey } }), // sandbox guests trust it ], })); ``` Bundles made before a key existed still load; they lack that key (`auditSigning`, `execTicketKey`, `sshCa`, `sandboxSshCa`, `commandSigning` are optional), and `execTicketIssuer` then fails with `INVALID_CONFIG`. `doctor` checks the bundle at `--pki`: `pki.unreadable` when it cannot be parsed, `pki.unseal` when it does not open with this deployment's secret, `pki.ca-expired` for an expired CA, `pki.ca-expiring` for one that expires within 30 days, and `pki.rotation` (info) while a command-signing root rotation is half done: a new root published but not signing yet (run `--stage switch` once every daemon has the trust version), or a former root still trusted (run `--stage drop` at least 24 hours after the switch). Doctor (and the support bundle) also compares the configured approver quorum with the published one (03bdbc9): `pki.quorum-unsigned` (warning) when it differs and its current approvers have not signed it for a higher epoch, since daemons holding approver keys would keep the old quorum (the fix names the exact `batch sign-approvers --epoch`, the published epoch plus 1, and `pki.approvers.epoch`; c6214dc), and `pki.quorum-change` (info) when the change is signed for a higher epoch or introduces the first set. ## Trust domain and SPIFFE IDs [#trust-domain-and-spiffe-ids] Every agent certificate carries exactly one SPIFFE URI SAN, first: ```text spiffe:///site//node/ ``` The trust domain is lower-case `a-z`, `0-9`, `.`, `-` and `_`, up to 255 characters, such as `compute.example.com`. The node id always comes from this SAN, never from a message field, so a daemon cannot speak for another node. The gateway checks the SPIFFE ID's trust domain against its configured one. Since d94f1f3 the SPIFFE ID may be followed by `urn:bc:` statements: what compute proved about the node's key when it issued the certificate. They are never identities. The node CA accepts at most four URI SANs, the SPIFFE ID and then only statements (`urn:bc::`, printable ASCII, `isNodeKeyStatement`), and no DNS SANs. The one statement today is written after a certified `tpm` join: ```text urn:bc:tpm-ak:: ``` It is the TPM name (hex: the 2-byte name algorithm, then a digest of 20 to 64 bytes) of the attestation key that certified the node key (`tpmAkUri`, `parseTpmAkUri` in `@better-compute/pki`; `pki.TPMAKName` in bcd). Every reader of a node identity takes the one URI SAN that is not a statement: `nodeSpiffeIdOf(uriSans)` in TypeScript (mTLS identity, proofs of possession, join responses, the gateway's renew relay) and `pki.NodeIdentity` in bcd, so a certificate with two SPIFFE IDs, or none, has no identity. The role travels in the subject as one `OU=bc-role:` attribute (1 to 32 of `a-z`, `0-9` and `-`, starting with a letter; a subject with two roles is malformed): `node`, `site`, `ramdisk` or `node-tenant` for agents, `gateway`, `internal` and `worker` for control-plane certificates. No certificate carries an OID under the private `2.25.` arc any more, because Go's `crypto/x509` refuses to parse such certificates, so bcd could not have read them. For one release, `parseCertificate` still reads the old role extension of certificates issued before the change; the OU wins when both are present, and Go accepts only the OU. Each role accepts only its own command kinds. A `node-tenant` certificate (a dedicated-root lease) runs as a node that gets no secrets and no destructive platform commands, and never attests. ## Node certificates [#node-certificates] JoinService issues a node certificate when a machine joins with a join token or another attestor, and records it in `node_certs`: | Column | Meaning | | --------------------------------- | ------------------------------------------------------------------------------------------------------ | | `serial` | The certificate serial | | `node_id`, `role` | Whose it is, and as what | | `identity_gen` | The node's identity generation when issued; a new generation revokes the old certificates | | `spki_sha256`, `der` | The key and certificate, so node signatures can be checked later | | `not_before`, `not_after` | Validity | | `revoked_at`, `revocation_reason` | Revocation | | `allowed_kinds` | When set, the only command kinds this certificate may receive; others are cancelled `kind-not-allowed` | Lifetimes: `node` and `site` certificates live 24 h; `ramdisk` certificates at most 2 h, whatever the profile asks. A daemon renews at about 60% of its lifetime with `Renew`, signing a proof of possession with its **current** key over the new CSR, its node id and its clock. The api checks the chain, revocation, the SPIFFE ID, a ±60 s window and a 120 s replay cache keyed by the signed statement. A certificate with a `urn:bc:tpm-ak:` statement also needs `RenewRequest.key_attestation`, that AK's certification of the new key; the statement is read from the presented leaf (byte for byte its `node_certs` row), never from the CSR, and copied into the new certificate (see [renewing a TPM-held key](/docs/guides/nodes/attestation#renewing-a-tpm-held-key)). A machine that lost its certificate re-joins with the `x509pop` attestor, which accepts a certificate of the current identity generation up to 7 days past its expiry. See [attestation](/docs/guides/nodes/attestation). ## Revocation [#revocation] A certificate is revoked by setting `node_certs.revoked_at`. Every path that ends an identity does it: `nodes.revokeIdentity`, node deletion, a reclaim, metal retire and a dedicated-root lease. Each gateway runs a revocation feed. It loads every revoked, unexpired certificate on start, then polls `node_certs.revoked_at` every second, re-reading 60 s before the newest revocation it has seen so out-of-order commits are still caught. After three failures in a row it polls every 5 s until one succeeds, and never stops the gateway. A serial the set learns about is refused at the next session open, and a live session gets a `RevocationNotice` and is closed. The target is a revoked node disconnected within 5 s. ## Signed intent [#signed-intent] With `pki.commandSigning` set, every command the queue writes is signed by a worker key: a P-256 key the command-signing root certifies for at most 24 h, with role `worker` (`OU=bc-role:worker`). A signer is valid only as a `worker` certificate chaining to a command-signing root; there is no command-signing extended key usage. Each process replaces its worker key after 60% of its lifetime. Daemons receive the roots in their trust bundle, first in `JoinResponse` and then in `Welcome`, signed by the JoinService key so a gateway can relay it but not change it. A daemon checks any signature that is present. Once it holds command-signing roots, a daemon requires a valid signature on every command whatever its kind, since an unsigned non-destructive kind could still act: bcd since 0d42de8 and the FakeDaemon since 12ccfa1 (review 7; pki's `VerifyIntentOptions.required`). Before, each daemon's kind table decided, requiring one only for destructive kinds. The same worker key signs desired state (v0.4). When the NodeSpec controller writes a new generation, it encodes the `DesiredStateBody` with pki's shared encoder (`storedDesiredState`, spec hash = SHA-256 of the encoded NodeSpec), signs it under purpose `desired-state` (`CommandSigner.signDesiredState`) and stores both in `nodes.spec_body` and `nodes.spec_signature` (`core/0021_signed_specs`) in the same `UPDATE` that bumps the generation. The gateway holds no key: it relays the stored bytes verbatim (`desiredStateFromStored`), and falls back to an unsigned rendering from `nodes.spec` (`desiredStateBodyFromJson`, byte-identical to the writer's) only when no current body is stored. `verifyDesiredState(desired, { roots, now, nodeId, required })` is the daemon-side check. Signed intent also changes how GPU bindings count: with `pki.commandSigning`, capacity's `attestedBindings` defaults to `pkiAllocationBindingCheck()`, which verifies each `AllocationBound` against the reporting node's own certificate in `node_certs` before the reservation goes active. See [attested bindings](/docs/guides/capacity/reservations#attested-bindings). Rotating the root takes three steps, gated on the fleet: 1. add the new root beside the old one (`pki.commandSigningRoots`); workers still sign under the old root, and the trust bundle's version goes up; 2. once every connected daemon reports that version, switch `pki.commandSigning` to the new root; 3. once nothing is signed under the old root any more, remove it; the version goes up again. **The version follows the content** (9c40cd5). A daemon takes a trust bundle only when its version is newer than the one it holds, so a changed bundle that kept its old version used to reach no daemon, silently. The deployment now records the digest of the bundle it publishes (the set of root hashes and the canonical quorum fields) and its version in `trust_bundle_state` (migrations `quorum/0003_trust_bundle_state` and `quorum/0004_trust_bundle_approver_set`, which also stores the published quorum). Each process settles the version once, at `initialize()` or before a gateway-only process first serves `JoinResponse.trust` or `Welcome.trust`: unchanged content keeps the recorded version, changed content (roots, approvers, tiers, the unbatched cap) takes the next one, and processes with the same bundle agree under a row lock. A bundle that was published before and has since been replaced never comes back as newer (0a31734, review 7): `trust_bundle_history` (migration `quorum/0006_trust_bundle_history`) keeps every published digest with its version, so a stale replica that restarts on an old configuration serves that bundle at its old version, which daemons ignore, and logs a warning, rather than publishing it as the next version and making daemons trust again, say, a command-signing root the new configuration removed. It does not change the record. Only an explicit `pki.trustVersion` above the recorded version publishes such a bundle again. `pki.trustVersion` is now a floor, not something you must bump: the published version is never below it. [Approver-set signatures](#changing-the-approver-set) no longer name a version: since c6214dc they sign the set's own epoch (`pki.approvers.epoch`), so they stay valid on every later version. `better-compute pki rotate-command-root --stage add|switch|drop` runs one step on the bundle and writes it back (sealed when `BETTER_COMPUTE_SECRET` is set), printing the roots and the bundle's trust version (with the content rule above, the version the deployment publishes may be higher, never lower): ```bash better-compute pki rotate-command-root --stage add # a new root beside the current one; trust version up # roll the bundle out to every process; daemons get the new version in Welcome.trust better-compute pki rotate-command-root --stage switch # workers sign under the new root; the old one stays trusted # at least 24 hours later (worker keys live at most 24 h) better-compute pki rotate-command-root --stage drop # the old root is no longer trusted; version up again ``` A step out of order is `INVALID_STATE`, and `doctor` reminds you of a rotation left half done (`pki.rotation`). So is a run with `BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1`: the command writes the bundle back, and only [`pki reseal`](#pki-reseal) vouches for an unauthenticated one. Every daemon reports the trust bundle version it holds: `Hello.trust_version` at session start, and `StatusReport.trust_version` again once it applies a newer bundle from `Welcome.trust` mid-session, so the acknowledgment does not wait for a reconnect. bcd and the FakeDaemon both send it. The gateway records it as `node_sessions.trust_version` (`core/0023_session_trust_version`; a status report only ever raises it). The `--stage switch` step reads that column and refuses with `INVALID_STATE` while any connected node is behind, naming a few (`1 of 2 connected node(s) have not applied it (node_old at 1)`), since those nodes would reject every command signed under the new root. Since the published version follows the content, the step gates on the version the deployment published for this bundle, not the bundle's own `trustVersion`: it recomputes the digest this bundle and configuration publish and refuses (`Not switching: the deployment still publishes trust version N without the new root`) until every process runs the `--stage add` bundle, then waits for daemons to hold that published version (03bdbc9). Connected means a session on a gateway, not ended, seen within the last 2 minutes; a node that connects later gets the current bundle in `Welcome` before any command. The check needs the deployment's configuration or `BETTER_COMPUTE_DATABASE_URL` and refuses without either; `--force` skips it. While a rotation is in progress, the `pki.rotation` finding of `doctor` and the support bundle says how many connected nodes already have the new version. `addTrustRoot`, `trustVersionAcknowledged` and `removeTrustRoot` in `better-compute/pki` implement the same gates for your own tooling. In production the design keeps this root in KMS or Vault transit; the bundle form is for development and small deployments. ### The destructive-batch quorum (v1.0) [#the-destructive-batch-quorum-v10] Signed intent stops a gateway from forging commands, but not a compromised api or worker, which holds a worker key. PLAN.md §13's answer for fleet-wide destructive work (reimage, wipe, reboot of many machines) is a quorum checked on the machine: destructive commands carry a `batch_id` in the worker-signed body, and run only once k distinct operator **approver keys**, pinned in the trust bundle and held off the control plane, have signed a `BatchApproval` over the Merkle root of the batch's scope. Each command ships with its inclusion proof, so the executing agent checks membership itself and neither the control plane nor a gateway can widen the batch. > **Built end to end; workflow leaves are unbound.** Built and tested end to end: the protocol seams, the `better-compute/pki` primitives, `pki.approvers` in the signed trust bundle with approver-set signatures for quorum changes, the server's `quorum` module (batches, their stored scopes, payloads and proofs, `commandBatches` approvals), the command queue's `createBatch` and `enqueue({ batchId })`, the gateway attaching each batched command's approval and inclusion proof (40be696), `commandBatches.create` and `commands.run({ batchId })` for batches over the API (46afe75), the approvers' `better-compute batch` commands, enforcement in both bcd and the FakeDaemon, and the producers: metal's workflows, `metal.bulk` and `metal.power`, the health reconciler's remediations and the MCP server's `propose_command_batch` all create their batches (see [who creates batches](#who-creates-batches)). What stays open: the batches built-in workflows create leave every leaf's payload unbound (any payload of its kind for its subject), because the bytes are only known later, and approvers see those leaves flagged; binding a leaf to the operation instead of its bytes is a post-v1.0 option in PLAN.md §13 (see the [roadmap](/docs/reference/roadmap)). **The scope.** A batch's scope is a tree with one leaf per distinct `(subject, kind, payload)`: `H(0x00 ‖ lp(batch_id) ‖ lp(subject) ‖ lp(kind) ‖ lp(payload_sha256))`, leaves ordered by hash, inner nodes `H(0x01 ‖ left ‖ right)` and an odd node promoted (RFC 6962 style; golden vectors in `packages/protocol/golden/batch-scope.json`). `payload_sha256` binds a leaf to exactly one operation, or is left empty to allow any payload of that kind for that subject; the agent accepts a proof from either the bound or the unbound leaf (77cdf4f; `command_batch_scope` and `command_batch_members` carry it, migration `quorum/0002_payload_digest`). A leaf may also carry its payload itself, `{ typeUrl, value }` (at most 64 KiB a leaf, and 8 MiB for all of a batch's leaves): its digest is then derived from those bytes (a `payloadSha256` given as well must match), and the payload is stored (migration `quorum/0007_leaf_payload`) so approvers can decode the image, firmware or power action they sign, not only a hex digest (2ca0ba4). The `typeUrl` must be `type.googleapis.com/` plus the kind's own payload type from the registry (fc2df7d, review 8), so a creator cannot label bytes as a harmless message. Every subject must name a machine of the batch's own tenant (fc2df7d): the node itself, the site agent in `bmc:/
`, or the target in `node:`. A subject naming another tenant's node is `NOT_FOUND` and a malformed one `INVALID_INPUT`, since approvers sign under the batch's tenant while the daemon checks only the leaf: otherwise a compromised worker could spend tenant A's approval on tenant B's machines. The subject is what the command acts on, computed by the executing agent from the signed body: | Kinds | Subject | | -------------------------------------------------------- | ------------------------------------------------------------------------- | | Kinds a node runs on itself | its `node_id` | | `bmc.*`, which a site agent runs against a machine's BMC | `bmc:` + the site agent's `node_id` + `/` + `BmcOperationRequest.address` | | `site.httpProxy` | `node:` + `HttpProxyRequest.target_node_id` | Without the BMC subject, "power-cycle 412 machines" would collapse into one `(site agent, bmc.power)` leaf, and approving it would let a compromised control plane power any machine behind that agent. BMC addresses are site-local, so the subject names the site agent too: an approval for a BMC at site A never admits a command at site B. **The wire.** `CommandBody.batch_id` is inside the worker signature; `Command.batch_approval` (`batch_id`, `scope_root`, `scope_size`, `not_after`, `signatures`) and `Command.batch_proof` (`index`, `siblings`) travel outside it, since approvers sign the approval themselves. The `TrustBundle` gains `approver_keys` (`kid`, `spki_der`), `approver_threshold`, `quorum_tiers` (`max_scope_size`, `threshold`), `unbatched_destructive_per_hour` and `approver_set_signatures` (see [changing the approver set](#changing-the-approver-set)), and `CommandRejectReason` gains `BATCH_UNAPPROVED` and `UNBATCHED_LIMIT`. The canonical quorum bytes approvers sign for a set change (`approverSetInput`: a u32 key count, then each key's length-prefixed `kid` and SPKI in `kid` order, the u32 threshold, the u32 tier count and each tier's pair, the u32 unbatched cap; all big-endian) have a golden vector beside `batch-scope.json`, `packages/protocol/golden/approver-set.json`, with the set digest and the signing input for an epoch, so bcd pins its encoding to the same bytes (26ab0fc). `approver_set_epoch` (field 8, c6214dc) numbers the quorum fields separately from the bundle's `version`. Approvals are signed under the purpose `batch-approval`, and the signed parts end with the deployment's trust domain, which every verifier takes from its own configuration: an approval signed for staging never admits a command in production. The server therefore needs `pki.join` (the trust domain) to accept approvals, and refuses the quorum with `INVALID_CONFIG` without it. **The primitives** in `better-compute/pki` (`packages/pki/src/quorum.ts`, and `approver-set.ts` for set changes): | Function | Does | | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `createApproverKey()`, `approverKeyOf(publicKey)` | A P-256 approver key pair, and its trust-bundle entry `{ kid, spkiDer }` with `kid` = base64url SHA-256 of the SPKI | | `batchSubject(body)`, `isBmcKind(kind)` | The subject above, from a command body; `INVALID_INPUT` when a relayed kind's payload is not its request or names no target | | `batchScope(batchId, entries)` | The tree over `{ subject, kind, payloadSha256? }` entries: `root`, `size` and `proofOf(subject, kind, payloadSha256?)` | | `verifyBatchProof({ batchId, subject, kind, payloadSha256?, proof, root, size })` | Walks a proof to the root; the size comes from the signed approval, never the command | | `signBatchApproval(approval, privateKey)` | One approver's `Signature` over `{ batchId, scopeRoot, scopeSize, notAfter, trustDomain }` | | `requiredApprovals(size, threshold, tiers)` | The first tier whose `maxScopeSize` covers the batch, else the threshold | | `verifyBatchApproval(approval, options)` | Unexpired, a 32-byte root, and enough valid signatures by distinct keys of the set (unknown kids, duplicates and a key listed under another's kid count for nothing); returns the approving kids | | `verifyBatchCommand(command, approval, proof, options)` | The agent's check: its own batch, an approval that holds, and its `(subject, kind)` in scope; `ACCESS_DENIED` with `details.reason` (`no-approval`, `other-batch`, `expired`, `too-few-approvals`, `not-in-scope`, ...) | | `approverSetOf(bundle)` | A `TrustBundle`'s quorum fields as an `ApproverSet` (`approverKeys`, `approverThreshold`, `quorumTiers`, `unbatchedDestructivePerHour`) | | `approverSetDigest(set)`, `sameApproverSet(a, b)` | SHA-256 of the canonical set bytes (`approverSetInput`), and whether two sets are the same quorum (keys in any order) | | `signApproverSet({ epoch, set, trustDomain }, privateKey)` | One current approver's `Signature` over a new set at one approver-set epoch of one deployment: purpose `approver-set`, parts `[uint64_be(epoch), approverSetDigest(set), trust_domain]` (4835f4f, c6214dc); an empty trust domain or an epoch below 1 is `INVALID_INPUT` | | `verifyApproverSetChange(current, next)` | Whether a holder of `current` (with its `epoch`) may take `next` (`next` carries `epoch`, `approverSetSignatures` and the verifier's own `trustDomain`, never the bundle's word for it): nothing is needed without current approver keys or with an unchanged set; otherwise `next.epoch` must be higher and the current threshold of distinct current keys must have signed. Returns the approving kids; `ACCESS_DENIED` with reason `stale-epoch`, `approver-set-unsigned`, `no-threshold` when the current threshold is 0, which admits no change, or `no-trust-domain`. On a refusal a daemon keeps its quorum and still applies the rest of the bundle | `TrustBundleInput` (for `signJoinResponse` and `signTrustBundle`) takes `approverSetSignatures` (`{ kid, sig }`) beside the quorum fields (6a7f1b7). **Publishing the keys.** `pki.approvers` puts the quorum in the join-key-signed trust bundle (`JoinResponse.trust` and `Welcome.trust`), so a gateway can relay it but not change it. The bundle's version follows the change by itself (see [the version follows the content](#signed-intent)). The first set needs nothing more, since a daemon without approver keys takes it as it comes (it only adds restrictions); any later change needs the current approvers (see [changing the approver set](#changing-the-approver-set)): ```ts title="better-compute.config.ts" export default defineConfig(() => ({ pki: { // nodeCa and join as above commandSigning: bundle.commandSigning, // required with approvers approvers: { keys: [aliceSpkiDer, bobSpkiDer, carolSpkiDer], // P-256 SPKI DER, distinct threshold: 2, // 1 to the number of keys tiers: [{ maxScopeSize: 5, threshold: 1 }], // small batches need fewer approvals unbatchedDestructivePerHour: 0, // the default: unbatched destructive commands are break-glass only }, }, })); ``` The options are validated at start (`INVALID_CONFIG`): `commandSigning` must be set, keys must be distinct P-256 public keys, thresholds whole numbers from 1 to the key count, tiers' `maxScopeSize` at least 1. Tiers must ascend strictly by `maxScopeSize`, their thresholds must never decrease, and none may exceed `threshold` (which applies to batches above every tier), so padding a scope with extra leaves never makes it cheaper to approve (d566cbe). A daemon cannot count fleet-wide, so per-daemon allowances for unbatched commands would let a compromised worker hit every node once; fleet-wide limits come from the approvers instead, for example a tier whose single signature comes from a rate-limited auto-approver service outside the api and workers. The [examples](/docs/operations/deployment) read the keys from `BETTER_COMPUTE_APPROVER_KEYS` (the `.pub` files of `batch keygen`, comma-separated), with `BETTER_COMPUTE_APPROVER_THRESHOLD` (required with keys, 1 to their count) and `BETTER_COMPUTE_UNBATCHED_PER_HOUR` (default 0), and refuse keys without a command-signing root in the bundle. **Batches on the server.** The built-in `quorum` module, always hosted, owns migration `quorum/0001_batches` (`command_batches`, `command_batch_scope`, `command_batch_members`, `command_batch_approvals`). `createCommandBatch(tx, { tenantId, scope, reason, notAfterMs? }, { now, quorum })` fixes a batch at creation: its root, size (at most 50,000 leaves) and every leaf's encoded `BatchProof` are stored, it ends after 24 hours by default, and it needs `requiredApprovals` for its size (0 without approver keys). A module reaches it through the command queue: `ctx.commands.createBatch(tx, { tenantId, scope, reason, notAfterMs?, createdBy? })` stores a batch with the trust bundle's quorum and answers `{ batchId, held, required, size }`, and `ctx.commands.enqueue(tx, { ..., batchId })` puts a command into it. The batch must belong to the node's owner and be pending or approved and unexpired, and the command's `(subject, kind)` (pki's `batchSubject`) must be in its scope (`INVALID_INPUT` otherwise); `batch_id` goes into the signed body, a `command_batch_members` row is written, and while a quorum that needs approvals has not approved the batch the command is held (`not_before` infinity, no poke). The commands API shows it (a299e96): a batched command's `CommandInfo` carries its `batchId`, and a held one `awaitingApproval: true` without `notBefore`, until the batch is approved. `enqueue` records which leaf the command uses: the one bound to its payload's SHA-256 when the scope has it, else the unbound leaf. Without approver keys a batch is recorded for audit and its commands go out at once. With approver keys and no break-glass allowance (`unbatchedDestructivePerHour` 0), `enqueue` refuses a destructive command without a `batchId` up front, since every daemon would refuse it `UNBATCHED_LIMIT`: `INVALID_STATE` with `details: { reason: 'batch-required', kind }`, naming `commands.createBatch` and the break-glass option (508728e). Destructive means the registry's `CommandKindSpec.destructive`, so plugin kinds count, and `node.reboot` is one. It is an early warning only; the daemon stays the judge. With an allowance, or without approver keys, unbatched destructive commands are enqueued as before. The gateway sends a batched command only once it can prove it (40be696): when the batch needs approvals, dispatch selects a member only after the batch is approved, joining the member to its scope leaf and the batch in SQL (so held rows never crowd the dispatch limit), and sends it with `Command.batch_approval` and the member's own leaf's `batch_proof`, decoded from the stored bytes. A `Hello` replay attaches them again. #### Who creates batches [#who-creates-batches] With approver keys set, every built-in producer of destructive commands puts them in a batch before it sends anything, so none of them trips `batch-required`: | Producer | Batch | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metal workflows (f66e236) | A workflow the reconciler starts outside a bulk batch gets one of its own: one node, the destructive leaves of the kinds it can send. `startWorkflow` records those kinds in `workflows.command_kinds` (every agent step's command, and every control-plane step's `sends`), and `workflows.batch_id` holds the batch (migration `provision/0006_workflow_batch`); the engine puts the batch id on the workflow's destructive commands only | | `metal.bulk` (f66e236) | One verb (`manage`, `inspect`, `provide`, `undeploy`, `rescue`, `retire`) on up to 10,000 nodes of one pool as one batch, whose scope covers the verb's whole chain of state profiles (inspect then commissioning, for example). Decided on the pool and on every node as `metal/`; the batch is parked on each node (`metal_nodes.batch_id`, migration `metal/0005_batch`) until the node rests. See [the metal API](/docs/reference/api/metal) | | `metal.power` (478025a) | A one-leaf batch, `bmc:/
` and `bmc.power`; the answer carries `batchId` and `held` | | Health remediation (e594a4f) | A reboot or GPU reset workflow gets a batch whose every leaf's subject is the node (`batchNodeWorkflow`); a workflow that relays `bmc.*` or `site.httpProxy` is refused there, since its owner must scope it | | MCP (171ad2f) | `propose_command_batch` creates a `node.reboot` batch over up to 1,000 nodes; `exec_command` takes its `batchId`. There is no approve tool: an agent only proposes, and people approve with their approver keys | | Your own code | `commandBatches.create` and `commands.run({ batchId })` over the API, or `ctx.commands.createBatch` and `enqueue({ batchId })` in a module | `metalBatchScope(tx, registry, nodeIds, kinds)` builds metal's scopes: the destructive kinds only, one leaf per node, and for a `bmc.*` kind one leaf per site agent of the node's site, since a site agent failover reroutes the command. The kinds come from `profileCommandKinds(profile, steps)` in provision, which relies on each control-plane step declaring what it `sends`: a plugin step that relays commands (Talos's `httpViaSite`) must declare `sends: ['site.httpProxy']`, or its commands fall outside the workflow's batch and are refused at enqueue (`INVALID_INPUT`, or `batch-required` when the workflow has no batch at all; see [steps and profiles](/docs/extending/steps-and-profiles)). Metal's leaves leave the payload unbound: the workflow builds its payloads later (a BMC credential rotated mid-workflow, user data rendered by an earlier step). Over the API, a caller creates a batch with `commandBatches.create` and runs commands in it with `commands.run({ ..., batchId })` (46afe75). `create` takes `tenantId`, a `reason` shown to approvers (at most 500 characters), the `scope` leaves (`{ subject, kind, payloadSha256?, payload? }`, `payloadSha256` as hex, `payload` as `{ typeUrl, value }` with `value` in base64; 1 to 50,000 of them), `notAfterSeconds` (60 s to 72 h, default 24 h) and an `idempotencyKey`. The batch id derives from the tenant, the caller's identity and the key, and migration `quorum/0005_batch_request` stores the request's digest, so a retry returns the same batch and approvers never sign two, while the same key reused for another request is `CONFLICT`. A new batch publishes `command.batch.created`. `commands.run` then checks the command's leaf against the scope and holds the command until approval, as `enqueue` does; with approver keys configured, even `node.reboot` needs such a batch. See [running commands](/docs/guides/nodes/commands). The `commandBatches` group (phase v1.0), decided on the tenant: | Method | Action | Does | | ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `create` | `compute:batches:create` | A batch over the given scope, answered like `get`; idempotent per caller and key | | `get`, `list` | `compute:batches:read` | A batch's state (`pending`, `approved`, `expired`, `cancelled`), root, size, required approvals, the approvals so far and its commands by state, and who created it: `createdBy`, plus `createdByAgent` and `delegationId` when an agent proposed it for that person (below); `list` pages by `state`, `limit`, `cursor` | | `scope` | `compute:batches:read` | The leaves (`leafIndex`, `subject`, `kind`, `payloadSha256`, empty for any payload, and the stored `payload` with `value` in base64 when its creator gave one), up to 5,000 a page, for approvers to review; check a payload against its digest before trusting it | | `members` | `compute:batches:read` | The batch's commands, each with its node, subject, kind, state, `reason` and `awaitingApproval` while held; filter `state`, paged by `limit` and `cursor` (`better-compute batch members`, the console's batch view) | | `approve` | `compute:batches:approve` | Takes one `signature` (`kid`, `sig`): it must verify alone for a known approver key over exactly this batch, is stored, and once the stored signatures meet the threshold the batch turns `approved`, its encoded `BatchApproval` is stored, its held commands become due and `command.batch.approved` is published | | `cancel` | `compute:batches:approve` | Ends a pending batch and cancels its held commands | **An agent's proposal reads as the agent's.** When the caller of `commandBatches.create` is an agent acting through a delegation (MCP's `propose_command_batch`, or a delegated API call), the batch records the agent and the delegation beside the person in `createdBy` (`command_batches.created_by_agent` and `delegation_id`, migration `quorum/0008_batch_agent`; f3c5c4e, security review 14 M3), since its `reason` is then the agent's text, not the person's. `batch approve` notes "Proposed by agent X for Y (delegation D): the reason is the agent's own text" before the approver signs, `batch show` answers the same words as `proposedBy`, and the console's Batches page shows them as a warning line (bd80209); server strings pass through the CLI's printable filter first. The IAM decision comes on top of the signatures, never instead of them. A pending batch past its end reads as `expired`, and the `quorum.batches` reconciler records that every minute and cancels its held commands; a deleted tenant's pending batches are cancelled. **The approvers' side.** The `better-compute batch` commands (b3fd712, 6a7f1b7; flags in [the CLI reference](/docs/reference/cli#quorum)) never trust the control plane's word for what a batch holds: | Command | Does | | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `batch keygen --out PATH` | A P-256 approver key (PKCS#8 PEM, mode 0600, never overwritten) and its public key at `PATH.pub`; prints the `kid`. With `BETTER_COMPUTE_APPROVER_PASSPHRASE` set, the private key is encrypted under it | | `batch list --tenant ID [--state STATE]` | `commandBatches.list`: reason, state, size, approvals had and needed, commands by state | | `batch show --tenant ID --batch ID [--scope-out FILE]` | Reads the batch, pages its scope and recomputes the root, like `approve`; `--scope-out` writes the verified scope with the batch's expiry and trust domain, the input of `batch sign` | | `batch approve --tenant ID --batch ID --key PATH` | Review, then sign and submit (below) | | `batch approve ... --signature FILE` | Submits a signature made offline by `batch sign`, after checking it signs this batch's id, root, size, expiry and trust domain (`SCOPE_MISMATCH` otherwise) | | `batch sign --key PATH --scope FILE` | Signs offline from a `--scope-out` file (or a `{ subject, kind }` list): the same review, then prints `{ kid, sig }` with the signed fields. `--not-after` and `--trust-domain` default to the file's. Signing a bare `--root` and `--size`, which it cannot check, needs `--unchecked` | | `batch create --tenant ID --reason TEXT --scope FILE [--hours N] [--key KEY]` | `commandBatches.create` from a JSON array of `{ subject, kind, payload }` leaves (def72c8): a `payload` given as `{ json }` is encoded as the kind's own payload message, or given as `{ typeUrl, value }`, so the leaf binds exactly that operation and approvers see it; leaves without one are warned about. `--hours` is 1 to 72 (default 24); the idempotency key is printed, and repeating it retries the same creation | | `batch members --tenant ID --batch ID [--state S] [--limit N] [--cursor ID]` | `commandBatches.members`: the batch's commands with node, subject, kind, state and `awaitingApproval` (45d217e) | | `batch cancel --tenant ID --batch ID` | `commandBatches.cancel` | | `batch sign-approvers --key PATH --epoch N ...` | Signs a change of the approver set for its epoch (see [changing the approver set](#changing-the-approver-set)) | `batch approve` refuses a batch that is not `pending`, pages the scope (up to 5,000 leaves a page) and recomputes the root from the listed `(subject, kind, payload digest)` leaves. Leaves that do not give the root and size the server asks approvers to sign, or more leaves than the size, are `SCOPE_MISMATCH` ("the control plane answered two different batches"); a page that does not advance, or an answer without a valid root and size, is `INVALID_RESPONSE`. It then shows the reason (and, for an agent's proposal, who proposed it for whom), every subject by kind (at most 12 per kind, then a count), and the approvals so far with the window's end, and signs the batch id, root, size, expiry and the deployment's trust domain. The review is enforced, not only shown: * Without `--yes` it stops after the summary and prints the root; `--root HEX` approves only if the batch still has the root you reviewed (`SCOPE_CHANGED` otherwise). * `--expect FILE` (subjects one per line, a JSON array of subjects, or `{ subject, kind }` commands) approves only a batch that holds exactly those; `SCOPE_UNEXPECTED` names what is missing or not expected. * Leaves that admit any payload are marked "any payload" and refused as `UNBOUND_SCOPE`, naming them, unless `--allow-unbound`. * Bound leaves' operations are shown decoded, grouped by the payload's digest (each line shows its first 12 hex digits), with long strings shortened in the middle so every field shows; `--operations-out FILE` writes every distinct operation in full with its subjects (`batch sign` and `batch approve`; b118616). A payload with fields this build cannot decode, or whose re-encoding does not give the bytes back, is refused (`payload-unknown-fields`), and a kind whose payload type the CLI does not know is opaque and needs `--allow-opaque`; it is never decoded as the type the server labels it. * A window that has closed, or that stays open longer than `--max-hours` (default 72, at most 720), is refused `INVALID_STATE`, since an approval admits its commands until then. * An approver key readable by others is refused (POSIX; `INVALID_STATE`), as ssh does; an encrypted one needs `BETTER_COMPUTE_APPROVER_PASSPHRASE` (`MISSING_ENV`). A key that already approved the batch is reported and nothing is signed. Once the batch holds the required approvals it is released and its commands run. **On the machine.** bcd and the FakeDaemon (`intent.quorum`, or the quorum of its signed trust bundle) enforce it the same way. With approver keys in the trust bundle, only kinds the daemon's own table marks destructive are checked: one with a `batch_id` runs only with a `batch_approval` of enough distinct approver signatures (the tier for its scope size, else the threshold, never fewer than one; a `kid` must be the SHA-256 of its key; duplicates count once), not past its `not_after` in server time, and a `batch_proof` that proves the command's own `(subject, kind)` leaf, the subject taken from the signed body; else it is `BATCH_UNAPPROVED`. Each leaf runs at most once, since an approval for N machines is N operations, not N kinds of operation to repeat. Both daemons key a used leaf by the hash of the leaf the proof matched, which holds the batch id, so another approval of the same batch id over a different tree cannot move a used leaf to a fresh index (the FakeDaemon keyed it by batch id and leaf index until c589c5d): another command on a used leaf is `BATCH_UNAPPROVED` (the FakeDaemon's reason is `leaf-used`), while a redelivery or re-execution of the same command keeps its leaf. The FakeDaemon enforces this (d566cbe), and so does bcd, which keeps the leaves that ran, with their command, in `data_dir/batch-leaves.json` until the approval's `not_after` (8b0f50c). One without a `batch_id` counts against a rolling-hour allowance (`unbatched_destructive_per_hour`, default 0), beyond which it is `UNBATCHED_LIMIT`; redeliveries do not count twice, and bcd keeps the count in `data_dir/unbatched-destructive.json` so a restart does not reset it. The verdicts apply after the signature check. bcd keeps the whole trust bundle in `identity.json`. Both daemons also hold these lines (3ff8131 and 8b0f50c in bcd, 6a7f1b7 in the FakeDaemon): * **A quorum change needs the current approvers.** Once a daemon holds approver keys, it refuses a change of any quorum field (keys, threshold, tiers or the unbatched cap; removal included) unless the current threshold of distinct current approvers signed the new fields, since the JoinService key alone could otherwise drop the approvers or swap in its own. A current threshold of 0 admits no change. Both apply one [epoch rule](#changing-the-approver-set) (c6214dc, 5fe7cbf, e171f0d): a refused set keeps the held quorum while the rest of the bundle (roots, version) applies; the FakeDaemon logs `trust.quorum-refused` with the reason. A daemon without approver keys takes the first set as it comes, or only its [pinned](#pinning-the-first-set) set, and its epoch never goes back. Both report the set they hold in `Hello` and `StatusReport` (`approver_set_epoch`, `approver_set_sha256`). * **The unbatched window counts on the local clock**, so a `Welcome` that moves server time forward cannot age the count out. Approvals still expire in server time. * **Nothing is used before admission.** A batch leaf or an unbatched slot is recorded only once the command is accepted and checked again at admission, so a forged, re-targeted or expired delivery carrying a public approval and proof cannot burn a leaf or the cap (the FakeDaemon's `claimQuorum`, review 6 L1); bcd also checks a deferred command's approval again when it is finally admitted. bcd alone refuses destructive commands when its stored bundle cannot be read, rather than switching the quorum off. bcd also treats a `site.httpProxy` other than a `GET` as a write (review 9): under a quorum it runs only under a leaf bound to its exact payload, never an unbound one, and never unbatched, whatever the hourly allowance; readiness polls, which are `GET`s, are unaffected. Both count `bmc.boot` and `bmc.rotate` as destructive, as metal's kind table does, so they are signed and batched like power actions. See [signed commands](/docs/daemon/delivery#signed-commands-and-attested-results). #### Changing the approver set [#changing-the-approver-set] The trust bundle is signed by the JoinService key, which the control plane holds, so on its own it must not be what changes the quorum. Adding or removing an approver, changing a threshold, a tier or the unbatched cap, or removing the quorum altogether therefore needs the **current** approvers' signatures over the new set (77cdf4f, 6a7f1b7, edbd63f). Since c6214dc (review 7 M1) they sign the set's own **epoch**, not the trust bundle version: `TrustBundle.approver_set_epoch` numbers the quorum fields, goes up with every change of the set, and stays put when roots rotate. `approver_set_signatures` are purpose `approver-set` over `[uint64_be(epoch), SHA-256(approverSetInput), trust_domain]`. Signing the version instead meant signatures were published only while that version was current, so a daemon offline during a set change met the next bundle (a root rotation, say) with the new set and no valid signatures, refused the whole bundle for good, and kept trusting rotated-out roots. The trust domain is the verifier's own (4835f4f, review 7): a staging set's published signatures would otherwise be replayable into production when the two share approver keys. One rule governs every change (5fe7cbf and e171f0d, review 10 M3; the rule in `common.proto`): a daemon holding set `H` at epoch `e` takes `(N, e')` only when `e' > e` and at least `H`'s threshold of `H`'s keys signed `(e', N, its own trust domain)`. That holds for `N == H` too: an unchanged set moves to a higher epoch only with its own signatures, and is otherwise kept silently at `e`. So a daemon that held `S@1` and missed `S -> T@2` and `T -> S@3` cannot later be moved to `T@2` by replaying that epoch's genuine signatures. The same set at the same epoch needs nothing; anything else is `stale-epoch` or `approver-set-unsigned`. `approver_set_signatures` may mix the previous set's signatures (the change) with the new set's own (which move daemons that already hold the new set to the new epoch); only held keys count. A daemon without keys takes the first set as it comes, or only its pinned one (below), never at or below an epoch it held, and keeps its epoch through a signed removal. When the quorum part is refused, the daemon keeps its quorum and still applies the rest of the bundle (roots, version). A daemon that missed two set changes whose keys do not overlap cannot verify the newer set with the keys it holds and needs re-provisioning (or re-pinning). The first set needs no signatures. On a fleet whose daemons already hold approver keys: ##### Pick the epoch [#pick-the-epoch] The new set's epoch is the published one plus 1. With the new quorum in the configuration, [`better-compute pki trust-version`](#pki-trust-version) prints it as `signQuorumForEpoch`, and doctor's `pki.quorum-unsigned` finding names it too. Any higher epoch works as well; an older one is never taken back. ##### Each current approver signs [#each-current-approver-signs] At least the current threshold of current approvers each run, with their own current key and the whole new set. Members of the new set may sign too: their signatures move daemons that already hold the new set to its epoch, and the command says when a key counts only as a current approver: ```bash better-compute batch sign-approvers --key alice.pem --epoch 2 \ --approvers alice.pem.pub,bob.pem.pub,dave.pem.pub --threshold 2 --tiers 5:1 --unbatched 0 ``` It validates the set (P-256 public keys from `batch keygen`, no key twice, a threshold from 1 to the key count, tiers ascending by size with thresholds that never decrease), says what it signs (a quorum of k of n with their kids, or, with no `--approvers` and `--threshold 0`, the REMOVAL of the quorum), notes when your own key is not in the new set, and prints `{ epoch, trustDomain, setSha256, set, signature: { kid, sig } }`. The trust domain is `--trust-domain`, else the PKI bundle's (`--pki`); without either it is a usage error. The examples refuse signature files made for another trust domain. ##### Publish the set with its signatures [#publish-the-set-with-its-signatures] Set the new quorum, its `epoch`, and the collected signatures as `pki.approvers.setSignatures`: ```ts title="better-compute.config.ts" pki: { commandSigning: bundle.commandSigning, approvers: { keys: [aliceSpkiDer, bobSpkiDer, daveSpkiDer], threshold: 2, tiers: [{ maxScopeSize: 5, threshold: 1 }], epoch: 2, // default 1; required with setSignatures: the epoch they sign setSignatures: [aliceSig, bobSig], // the { kid, sig } of sign-approvers (sig base64url or bytes) }, }, ``` `pki.approvers.epoch` is a whole number of at least 1 (default 1), `setSignatures` without it is `INVALID_CONFIG`, and each signature must be `{ kid, sig }` with a 64-byte P-256 signature. The bundle's version still follows its content; `JoinResponse.trust` and `Welcome.trust` carry `approver_set_epoch` and the set's signatures on every version while the set is current, so a later root rotation keeps them and a daemon that missed a version still verifies the set. `trust_bundle_state` records the published set with its epoch. A quorum change without valid signatures for a higher epoch is still published, since daemons without approver keys take it, but each process logs a warning naming `batch sign-approvers --epoch `, and daemons that hold keys keep their old quorum until a signed set reaches them. To fix it, have the current approvers sign that epoch and set `pki.approvers.epoch` and `setSignatures`. `better-compute doctor` verifies `setSignatures` as a daemon does (the published set's threshold, the epoch, the deployment's trust domain), so signatures that do not verify are `pki.quorum-unsigned` with the reason (b118616). **Nodes that hold another set.** A daemon can take a bundle's version while refusing its quorum part, so the trust version alone does not show which approvers a node enforces. The gateway records the set each daemon reports holding in `node_sessions.approver_set_epoch` and `approver_set_sha256` (migration `core/0026_session_approver_set`, fa39201): * `Hello` writes the reported set as it comes. * A `StatusReport` with `trust_version` above 0 (the acknowledgement of a newer bundle) writes it only when that version is at least the one recorded, so a late, older report never pairs an old set with a newer version. * A daemon that holds no approver keys (an empty digest) is recorded as `NULL` in both columns. `better-compute doctor` and [`pki trust-version`](#pki-trust-version) read those columns for the connected nodes (a session on a gateway, not ended, seen within the last 2 minutes) and compare them with the published set (6612496, 08eb62e). While the published bundle has approver keys, a node lags when its digest is `NULL` (it enforces no quorum at all, the worst lag), differs from the published set's, or its epoch differs. Without published keys, any node that still holds a set lags. Doctor reports `pki.quorum-lag` (a warning, only when a published approver set exists) as "N of M connected node(s) enforce another approver set than the published one (epoch E)", followed by up to five nodes with the epoch each holds (0 for a node without keys). `pki trust-version` prints the count as `approverSetBehind` and those nodes as `approverSetSample` (`nodeId`, `epoch`). Both leave the check out before `core/0026` has run. The fix is to check the set's signatures, or to re-pin nodes that missed several changes. The examples read the signatures from `BETTER_COMPUTE_APPROVER_SET_SIGNATURES`: the JSON files `batch sign-approvers` printed, comma-separated. They must all sign one epoch and exactly the quorum `BETTER_COMPUTE_APPROVER_KEYS`, `BETTER_COMPUTE_APPROVER_THRESHOLD` and `BETTER_COMPUTE_UNBATCHED_PER_HOUR` configure (the examples set no tiers), or the example refuses to start; that epoch becomes `pki.approvers.epoch`. `BETTER_COMPUTE_TRUST_VERSION` sets the trust bundle version's floor, independently. #### Pinning the first set [#pinning-the-first-set] A daemon without approver keys (every new or reimaged node) takes the first set its join-key-signed bundle carries, so a compromised api could hand it an empty set or its own key at k = 1, after which the operators' real set would need the attacker's signature. The first set can be pinned outside the api (9b7a199, review 7 M2): `better-compute pki trust-version` prints `approverSetSha256` (SHA-256 of `approverSetInput`), computed from this configuration's approver keys rather than from the database the api writes, with `publishedApproverSetSha256` beside it while the two differ (b118616, review 10 M4), and `daemon install-script --approver-set-sha256 HEX` writes it into the node's `config.toml` as `approver_set_sha256` (an upgrade run without the flag keeps an existing pin, b6d401f). A daemon with that pin and no keys takes only a matching set (`pin-mismatch` otherwise, while the rest of the bundle still applies) and refuses destructive commands `BATCH_UNAPPROVED` (the FakeDaemon's reason is `awaiting-pinned-set`) until it holds one, so it fails closed. bcd (`daemon/internal/pki/batch.go`, `agent/quorum.go`) and the FakeDaemon (`intent.approverSetSha256`) both implement it. #### Turning the quorum on [#turning-the-quorum-on] PLAN.md §13 records what happens when a deployment upgraded from v0.5 sets `pki.approvers` for the first time, and `tests/sim/upgrade.test.ts` proves it (80195df): no allocation moves, no VM reboots, and reservations stay active. ##### Upgrade first, then set the approvers [#upgrade-first-then-set-the-approvers] Run the v1.0 upgrade as usual (the quorum migrations as expand migrations, then workers, gateways and daemons), then set `pki.approvers` and restart the workers and roll the gateways again. Setting approvers moves no NodeSpec generation. ##### Cancel pending destructive commands [#cancel-pending-destructive-commands] A destructive command queued before the switch is still sent afterwards, but a daemon that holds the quorum by then rejects it `UNBATCHED_LIMIT`, so it never runs. To have nothing destructive in flight at the switch, cancel pending destructive commands first (`commands.cancel`). ##### Let the bundle roll out [#let-the-bundle-roll-out] The trust bundle's version goes up exactly once when approvers are set. Daemons take the first set without signatures, or only the set they were pinned to; check with `pki trust-version` that none is behind. ##### Batch destructive work from now on [#batch-destructive-work-from-now-on] A new unbatched destructive command is refused at enqueue (`INVALID_STATE`, reason `batch-required`) unless `unbatchedDestructivePerHour` allows it. The built-in producers batch their own work (see [who creates batches](#who-creates-batches)). ## Exec tickets and the SSH CA [#exec-tickets-and-the-ssh-ca] The exec ticket key signs ES256 JWTs (`typ: bc-exec+jwt`) that live at most 60 s, name one node as audience (`aud: node:`), and carry a single-use `jti`. `tunnels.open` issues them after Better IAM allowed the call; the gateway and then the daemon verify each ticket against the JWKS in the node's NodeSpec. Every api replica and the gateway derive the same `kid` from the key, and `rotate()` keeps the previous key listed. The SSH CA issues OpenSSH user certificates for break-glass access, valid at most 1 h and never beyond the requesting session, with the principal `@` so a certificate opens only the node it names. Every issuance is a row in `ssh_certificates`; `nodes.revokeSshCert` adds the serial to the node's key revocation list in its next NodeSpec. See [SSH](/docs/guides/access/ssh) and [tunnels](/docs/guides/access/tunnels). ## The sandbox SSH CA [#the-sandbox-ssh-ca] The bundle's `sandboxSshCa` (v0.5) is a second OpenSSH CA, used only by `sandboxes.exec`. It signs user certificates of 60 seconds to 15 minutes, never past the caller's session or the sandbox's expiry, for the one principal `@` (for a sandbox restored from a memory snapshot, the instance the memory was captured on, since the resumed guest keeps that principal). It is kept apart from the break-glass CA so neither certificate opens what the other is for: nodes never trust the sandbox CA, and sandbox guests never trust the break-glass CA. * **Signing.** Pass it as `tunnels.sandboxSsh` (`tunnelsOptions(bundle, …)` does). Without it, `sandboxes.exec` is `UNSUPPORTED`. The tunnels module refuses a `sandboxSsh` whose key is the break-glass `ssh` CA's (`INVALID_CONFIG`: "tunnels.sandboxSsh must be a different CA from tunnels.ssh"), so the split cannot be undone by configuration. * **Trust.** Pass its public key as `vps({ sandboxExec: { caPublicKey } })`. vps writes it into every sandbox instance's user data as `TrustedUserCAKeys`, with an `AuthorizedPrincipalsFile` holding that instance's own principal. vps and the tunnels module each refuse a key that is not one line (an ed25519 or ECDSA P-256 key for vps), since a second line would be a second trusted CA; the bundle loader refuses one too. * **Older bundles.** `pki add-sandbox-ca` adds it; `pki show` prints it as `sandboxSshCaPublicKey`. * **Revocation.** Guests have no agent and receive no revocation list, so the short lifetime and tunnel re-authorization are the controls. See [sandboxes](/docs/guides/vps/sandboxes#run-code-inside). * **Host certificates.** Every `SshCertificateAuthority` can also sign OpenSSH host certificates: `issueHostCertificate({ publicKey, principals, keyId, validBefore })` returns `{ certificate, serial, validBefore }` with no options or extensions, for principals that are plain names (letters, digits, `.`, `_`, `-`; never a list or a wildcard) and a lifetime of at most `maxHostTtlSeconds` (default and maximum 30 days). Wire the sandbox CA's `issueHostCertificate` into `vps({ sandboxExec: { certifyHostKey } })` and sandbox guests present host keys it certified, which `sandbox exec` pins with `@cert-authority` instead of accepting them on first use; see [how the guest trusts it](/docs/guides/vps/sandboxes#how-the-guest-trusts-it). ## The audit key [#the-audit-key] With the bundle's `auditSigning` key, `auditModule({ signer })` seals `audit_events` into signed, hash-chained, per-tenant Merkle checkpoints every hour. `better-compute audit verify --tenant ID` checks a tenant's trail against the key's pin, taken from `--pin`, `BETTER_COMPUTE_AUDIT_PIN` or the bundle's public half, and exits `AUDIT_TAMPERED` on any problem. See [audit](/docs/guides/events/audit). ## Workload identity [#workload-identity] Workload identity (v0.5, optional) gives hosted VPS instances short-lived OIDC tokens. Its key is not part of the bundle: `createWorkloadIssuer({ issuer, defaultAudience, keys? })` from `@better-compute/pki` holds its own P-256 keys, and you pass the issuer as `betterCompute({ workloadIdentity: { issuer } })`. * **Tokens.** ES256 JWTs with `iss`, `sub = "t::instance:"`, `aud` (the default audience unless the request names an allowed one), `iat`, `nbf`, `exp`, `jti`, `tenant_id` and `instance_id`, living 60 to 900 seconds. Extra claims may not shadow those. * **Publication.** The issuer URL must be https on port 443, without query. `better-compute serve` answers `GET` and `HEAD` on `/.well-known/openid-configuration` and `/.well-known/jwks.json`, public, cached 5 minutes, with `access-control-allow-origin: *`; the JWKS is read per request, so an added key shows at once. The issuer URL has to reach that host. * **Rotation**, in three steps so no token ever fails verification: `addKey()` publishes a new key while the old one still signs; once verifiers have re-read the JWKS, `promote(kid)` signs with it; `retire(kid)` drops the old key, refused until its last token has expired (15 minutes plus a minute after it stopped signing). `exportKeys()` returns the keys as PKCS#8 PEM with their state, for storage; handle them as secrets. The host asks for a token over its session with a `WorkloadTokenRequest` signed by its node key (purpose `workload-token`), and the answer is sealed to a one-time key (JWE `typ: bcworkload+jwe`, binding the request id, the instance and the expiry). `createWorkloadTokenRequest` and `openWorkloadTokenResponse` are the host's half, `openWorkloadTokenRequest`, `sealWorkloadTokenResponse` and `refuseWorkloadTokenRequest` the broker's. bcd carries the host's half in Go (`NewWorkloadTokenRequest` and `OpenWorkloadTokenResponse` in `daemon/internal/pki`, checked against fixtures the TypeScript side produces), and its metadata service (`daemon/internal/imds`) calls it for the instances of whichever runtime bcd runs; neither runtime of host builds (`libvirt-kvm`, `cloud-hypervisor`) has run on real hardware yet, so only fakehw builds' instances have been served end to end. What the broker checks is in [VPS instances](/docs/guides/vps/instances#workload-identity). ## Rotation and expiry [#rotation-and-expiry] | What | How it is replaced today | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Node certificates | Automatically: daemons renew at about 60% of their 24 h lifetime | | Worker signing keys | Automatically, at 60% of their 24 h lifetime | | Gateway serving certificates | `pki issue-gateway` before the 30 days run out; gateways with `host.gateway.tlsFiles` hot-reload it, others need a restart one at a time | | Command-signing root | `pki rotate-command-root`, the three steps above | | Exec ticket key | `rotate()` on the issuer; the previous key stays in the JWKS | | SSH CA, sandbox SSH CA | Not automated. A new sandbox CA reaches only instances created after it, since the key is in their user data | | A bundle sealed in the former format | Once: `pki reseal` prints its pins, then `pki reseal --accept-unauthenticated` after comparing them (see [pki reseal](#pki-reseal)) | | Workload issuer keys | `addKey()`, `promote(kid)`, then `retire(kid)` once the old key's tokens expired | | Sealed secrets | `localSealedSecretStore` with several keys, then `reseal()` (see [configuration](/docs/operations/configuration#the-deployment-secret)) | | Node, serving and internal CAs (1 year) | `pki renew-ca --ca node` (or `serving`, `internal`) before they expire (`doctor` warns 30 days ahead); see [pki renew-ca](#pki-renew-ca) | | JoinService keys | Not automated. Daemons pin them, so new keys mean new pins in every machine's configuration. | > **Losing the bundle.** A bundle that is lost, or that no longer opens because the secret changed, cannot be recreated: a new `pki init` makes new CAs and new pins, and every node has to rejoin. Back up the bundle and the secret, and keep the offline root key apart from both.