PKI

Better IAM identifies people and AI agents. Machines are identified by Better Compute's own PKI: a set of local certificate authorities and keys that one deployment owns, created once with better-compute pki init and loaded by every process at start. Everything is ECDSA P-256 with SHA-256, except the two SSH CAs, which are Ed25519 by default, and the package needs nothing beyond node:crypto.

The design rule is that no key does two jobs. A CA that signs agent certificates cannot sign a gateway's server certificate, and the key that signs JoinResponse is not the one join tokens are sealed to.

The keys

Subordinate CAs never outlive the root: a CA certificate's notAfter is capped at the root's. Leaf lifetimes are enforced by the CA itself, whatever a caller asks for.

The gateway holds none of these keys in its own code: it relays Join, Renew and secret requests to JoinService and never signs anything. See security for what that means for the process that runs it.

The pki commands

pki init

export BETTER_COMPUTE_SECRET=$(openssl rand -base64 48)   # the same for every process
better-compute pki init --trust-domain compute.example.com

It creates every key above and writes one JSON file, mode 0600, never overwriting an existing one. The private keys are sealed with AES-256-GCM under a key derived (HKDF-SHA256) from BETTER_COMPUTE_SECRET, with the trust domain in the authenticated data. The sealed payload also carries a SHA-256 over every public field (certificates, trust anchors, the trust version), so whoever can write the file but lacks the secret cannot add a command-signing root or pin a trust version: loading refuses a sealed bundle whose public parts changed (INVALID_CONFIG; restore it from a backup), while reformatting the JSON is harmless. Any CLI write-back re-seals in the current format. pki init prints the pins, like pki show.

Without the secret and without --plaintext it refuses with MISSING_ENV. Keep the --offline-root file off the servers: it is needed only to replace a CA.

pki show

pki show reads the public half of the bundle. With BETTER_COMPUTE_SECRET it first checks that half against the seal and prints authenticated: true; without the secret it still prints the pins, with authenticated: false and a warning that whoever can write the file chooses them. audit verify and daemon install-script read pins the same way. It prints:

Pins have bcd's form, sha256/<base64 of SHA-256(SPKI)>. better-compute daemon install-script reads the bundle and writes the serving CA pin and the JoinService pins and key into each machine's configuration.

pki issue-gateway

better-compute pki issue-gateway --san gw.eu-1.compute.example.com --out /etc/better-compute/tls --days 14

It generates a P-256 key and issues a serving certificate from the bundle's serving CA. It writes NAME.key (mode 0600) and NAME.crt (the certificate followed by the serving CA, stopping below the root) into --out, replacing earlier files atomically.

pki add-sandbox-ca

better-compute pki add-sandbox-ca

Bundles from pki init already hold a sandbox SSH CA. For a bundle made before sandboxes, this creates one (Ed25519, comment <trust domain> sandbox ssh ca), writes the bundle back (sealed when BETTER_COMPUTE_SECRET is set; an offline root stays out) and prints sandboxSshCaPublicKey. A bundle that already has one is INVALID_STATE. It writes the bundle back, so it refuses BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1 (INVALID_STATE): re-seal a former-format bundle with pki reseal first. Roll the bundle out to every api replica, and give vps the public key as sandboxExec.caPublicKey. The only flag is --pki.

pki reseal

better-compute pki reseal                            # an unauthenticated bundle: prints its pins and stops
better-compute pki reseal --accept-unauthenticated   # after comparing them with your records

It rewrites the bundle sealed with BETTER_COMPUTE_SECRET (required, at least 32 characters; MISSING_ENV otherwise) in the current format, where the seal also covers every certificate and pin, so a file changed without the secret no longer loads. It keeps every key and certificate: nothing is rotated.

• An authenticated bundle (already sealed in the current format under this secret) is simply resealed.
• An unauthenticated bundle (sealed in the former format, or unsealed while BETTER_COMPUTE_SECRET is set) is not sealed at first: resealing would vouch for whatever the file holds. The command prints the pins it would vouch for, the same fields as pki show with authenticated: false, and stops with INVALID_STATE ("nothing vouches for the pins above"). Compare them, and the certificates digests (which also reveal a certificate appended behind a pinned one), with your install scripts, join tokens and pinned CAs, then run it again with --accept-unauthenticated.

After sealing it prints the pins it now vouches for, with authenticated: true. It needs no BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED.

pki renew-ca

better-compute pki renew-ca --ca node
better-compute pki renew-ca --ca serving --root-key /media/usb/root-ca.key

It issues a new certificate for the node, serving or internal CA from the root, for the same key and subject, and writes the bundle back (sealed when BETTER_COMPUTE_SECRET is set). Certificates the CA already issued stay valid and the serving CA's SPKI pin does not change, so nothing is re-pinned and no node rejoins. Roll the bundle out to every process before the old certificate expires. It prints the CA, the new notAfter and serial, and for serving the unchanged pin. Like every command that writes the bundle back, it refuses BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1 with INVALID_STATE.

pki trust-version

better-compute pki trust-version

It reads the trust bundle the deployment last published (trust_bundle_state; its version follows its content) through the deployment's configuration and compares it with this configuration and PKI bundle (03bdbc9). It prints version (the published one), configurationMatches (whether this bundle's roots and the configured quorum publish the same bundle), nextVersion (what this configuration would publish: the same version, or the next one), approverSetEpoch (the published approver set's epoch), approverSetSha256 (this configuration's set digest, for pinning the first set, with publishedApproverSetSha256 while they differ), approverSetBehind and approverSetSample (connected nodes enforcing another set, as the gateway records it, a node without keys included; see nodes that hold another set; 6612496, 08eb62e), quorumChanged and, when it did, signQuorumForEpoch (the epoch to sign the change for with batch sign-approvers, one above the published one; c6214dc), and how many connected daemons there are and how many are behind that version. Before any process has published a bundle it is NOT_FOUND. --pki PATH names the bundle.

Reloading gateway certificates

A running gateway picks a reissued certificate up without a restart when it reads it from files. Point host.gateway.tlsFiles at them (and host.gateway.internal.tlsFiles at the internal-CA pair of the control API):

export const host = defineHost(() => ({
  gateway: {
    region: 'eu-1',
    tls: {}, // other listener options; key, cert and ca come from the files
    tlsFiles: {
      key: '/etc/better-compute/tls/gateway.key',
      cert: '/etc/better-compute/tls/gateway.crt',
      ca: '/etc/better-compute/tls/node-ca.pem', // node CA roots, for client verification
      intervalMs: 60_000, // the default
    },
  },
}));

serve reads the files at start over tls and then checks their modification time and size every intervalMs. A changed set is loaded into a new secure context first and swapped onto the daemon listener and the WebSocket fallback (or the control API): new handshakes get the renewed certificate, and open sessions keep theirs, so no daemon is disconnected by a rotation. A half-written or mismatched pair is reported (Gateway serving certificate not reloaded: …) and the listener keeps its certificate; the next change is tried again. Without tlsFiles, the certificate in tls is read once at start, so after reissuing, restart gateways one at a time.

Outside serve, serveGateway, serveGatewayWs and serveInternal from better-compute/gateway return setTls({ key, cert, ca? }), which applies new material over the listener's original TLS options (protocol versions and the rest stay) and throws INVALID_CONFIG, changing nothing, when the key does not match its certificate. watchTlsFiles({ key, cert, ca? }, apply, { intervalMs?, onError?, onReload? }) polls the files and calls apply with a changed set, for example (tls) => served.setTls(tls).

The bundle in code

The bundle is a JSON document (format: "better-compute-pki", version: 1) with the certificates and public keys in the clear and the private keys either sealed or, with --plaintext, as PEM.

Wire it into the options and the gateway:

const bundle = await loadPkiBundle(env.BETTER_COMPUTE_PKI ?? '.better-compute/pki.json', env);

export default defineConfig(() => ({
  // …
  pki: {
    nodeCa: bundle.nodeCa, // the gateway role's trust anchor and revocation set
    join: pkiJoinCrypto({
      trustDomain: bundle.trustDomain,
      nodeCa: bundle.nodeCa,
      signingKey: bundle.joinSigning,
      encryptionKey: bundle.joinEncryption.privateKey,
    }),
    commandSigning: bundle.commandSigning, // v0.2 signed intent
  },
  // The ticket issuer must be baseURL's origin: the api accepts only tickets from it.
  // Equivalent to { issuer: execTicketIssuer(bundle, …), ssh: bundle.sshCa, sandboxSsh: bundle.sandboxSshCa }.
  tunnels: tunnelsOptions(bundle, { issuer: 'https://compute.example.com' }),
  modules: [
    auditModule({ signer: bundle.auditSigning }),
    vps({ images, sandboxExec: { caPublicKey: bundle.sandboxSshCa!.publicKey } }), // sandbox guests trust it
  ],
}));

Bundles made before a key existed still load; they lack that key (auditSigning, execTicketKey, sshCa, sandboxSshCa, commandSigning are optional), and execTicketIssuer then fails with INVALID_CONFIG.

doctor checks the bundle at --pki: pki.unreadable when it cannot be parsed, pki.unseal when it does not open with this deployment's secret, pki.ca-expired for an expired CA, pki.ca-expiring for one that expires within 30 days, and pki.rotation (info) while a command-signing root rotation is half done: a new root published but not signing yet (run --stage switch once every daemon has the trust version), or a former root still trusted (run --stage drop at least 24 hours after the switch). Doctor (and the support bundle) also compares the configured approver quorum with the published one (03bdbc9): pki.quorum-unsigned (warning) when it differs and its current approvers have not signed it for a higher epoch, since daemons holding approver keys would keep the old quorum (the fix names the exact batch sign-approvers --epoch, the published epoch plus 1, and pki.approvers.epoch; c6214dc), and pki.quorum-change (info) when the change is signed for a higher epoch or introduces the first set.

Trust domain and SPIFFE IDs

Every agent certificate carries exactly one SPIFFE URI SAN, first:

spiffe://<trust-domain>/site/<siteId>/node/<nodeId>

The trust domain is lower-case a-z, 0-9, ., - and _, up to 255 characters, such as compute.example.com. The node id always comes from this SAN, never from a message field, so a daemon cannot speak for another node. The gateway checks the SPIFFE ID's trust domain against its configured one.

Since d94f1f3 the SPIFFE ID may be followed by urn:bc: statements: what compute proved about the node's key when it issued the certificate. They are never identities. The node CA accepts at most four URI SANs, the SPIFFE ID and then only statements (urn:bc:<kind>:<value>, printable ASCII, isNodeKeyStatement), and no DNS SANs. The one statement today is written after a certified tpm join:

urn:bc:tpm-ak:<nameAlg>:<digest>

It is the TPM name (hex: the 2-byte name algorithm, then a digest of 20 to 64 bytes) of the attestation key that certified the node key (tpmAkUri, parseTpmAkUri in @better-compute/pki; pki.TPMAKName in bcd). Every reader of a node identity takes the one URI SAN that is not a statement: nodeSpiffeIdOf(uriSans) in TypeScript (mTLS identity, proofs of possession, join responses, the gateway's renew relay) and pki.NodeIdentity in bcd, so a certificate with two SPIFFE IDs, or none, has no identity.

The role travels in the subject as one OU=bc-role:<role> attribute (1 to 32 of a-z, 0-9 and -, starting with a letter; a subject with two roles is malformed): node, site, ramdisk or node-tenant for agents, gateway, internal and worker for control-plane certificates. No certificate carries an OID under the private 2.25.<uuid> arc any more, because Go's crypto/x509 refuses to parse such certificates, so bcd could not have read them. For one release, parseCertificate still reads the old role extension of certificates issued before the change; the OU wins when both are present, and Go accepts only the OU. Each role accepts only its own command kinds. A node-tenant certificate (a dedicated-root lease) runs as a node that gets no secrets and no destructive platform commands, and never attests.

Node certificates

JoinService issues a node certificate when a machine joins with a join token or another attestor, and records it in node_certs:

Lifetimes: node and site certificates live 24 h; ramdisk certificates at most 2 h, whatever the profile asks. A daemon renews at about 60% of its lifetime with Renew, signing a proof of possession with its current key over the new CSR, its node id and its clock. The api checks the chain, revocation, the SPIFFE ID, a ±60 s window and a 120 s replay cache keyed by the signed statement. A certificate with a urn:bc:tpm-ak: statement also needs RenewRequest.key_attestation, that AK's certification of the new key; the statement is read from the presented leaf (byte for byte its node_certs row), never from the CSR, and copied into the new certificate (see renewing a TPM-held key). A machine that lost its certificate re-joins with the x509pop attestor, which accepts a certificate of the current identity generation up to 7 days past its expiry. See attestation.

Revocation

A certificate is revoked by setting node_certs.revoked_at. Every path that ends an identity does it: nodes.revokeIdentity, node deletion, a reclaim, metal retire and a dedicated-root lease.

Each gateway runs a revocation feed. It loads every revoked, unexpired certificate on start, then polls node_certs.revoked_at every second, re-reading 60 s before the newest revocation it has seen so out-of-order commits are still caught. After three failures in a row it polls every 5 s until one succeeds, and never stops the gateway. A serial the set learns about is refused at the next session open, and a live session gets a RevocationNotice and is closed. The target is a revoked node disconnected within 5 s.

Signed intent

With pki.commandSigning set, every command the queue writes is signed by a worker key: a P-256 key the command-signing root certifies for at most 24 h, with role worker (OU=bc-role:worker). A signer is valid only as a worker certificate chaining to a command-signing root; there is no command-signing extended key usage. Each process replaces its worker key after 60% of its lifetime. Daemons receive the roots in their trust bundle, first in JoinResponse and then in Welcome, signed by the JoinService key so a gateway can relay it but not change it. A daemon checks any signature that is present. Once it holds command-signing roots, a daemon requires a valid signature on every command whatever its kind, since an unsigned non-destructive kind could still act: bcd since 0d42de8 and the FakeDaemon since 12ccfa1 (review 7; pki's VerifyIntentOptions.required). Before, each daemon's kind table decided, requiring one only for destructive kinds.

The same worker key signs desired state (v0.4). When the NodeSpec controller writes a new generation, it encodes the DesiredStateBody with pki's shared encoder (storedDesiredState, spec hash = SHA-256 of the encoded NodeSpec), signs it under purpose desired-state (CommandSigner.signDesiredState) and stores both in nodes.spec_body and nodes.spec_signature (core/0021_signed_specs) in the same UPDATE that bumps the generation. The gateway holds no key: it relays the stored bytes verbatim (desiredStateFromStored), and falls back to an unsigned rendering from nodes.spec (desiredStateBodyFromJson, byte-identical to the writer's) only when no current body is stored. verifyDesiredState(desired, { roots, now, nodeId, required }) is the daemon-side check.

Signed intent also changes how GPU bindings count: with pki.commandSigning, capacity's attestedBindings defaults to pkiAllocationBindingCheck(), which verifies each AllocationBound against the reporting node's own certificate in node_certs before the reservation goes active. See attested bindings.

Rotating the root takes three steps, gated on the fleet:

1. add the new root beside the old one (pki.commandSigningRoots); workers still sign under the old root, and the trust bundle's version goes up;
2. once every connected daemon reports that version, switch pki.commandSigning to the new root;
3. once nothing is signed under the old root any more, remove it; the version goes up again.

The version follows the content (9c40cd5). A daemon takes a trust bundle only when its version is newer than the one it holds, so a changed bundle that kept its old version used to reach no daemon, silently. The deployment now records the digest of the bundle it publishes (the set of root hashes and the canonical quorum fields) and its version in trust_bundle_state (migrations quorum/0003_trust_bundle_state and quorum/0004_trust_bundle_approver_set, which also stores the published quorum). Each process settles the version once, at initialize() or before a gateway-only process first serves JoinResponse.trust or Welcome.trust: unchanged content keeps the recorded version, changed content (roots, approvers, tiers, the unbatched cap) takes the next one, and processes with the same bundle agree under a row lock. A bundle that was published before and has since been replaced never comes back as newer (0a31734, review 7): trust_bundle_history (migration quorum/0006_trust_bundle_history) keeps every published digest with its version, so a stale replica that restarts on an old configuration serves that bundle at its old version, which daemons ignore, and logs a warning, rather than publishing it as the next version and making daemons trust again, say, a command-signing root the new configuration removed. It does not change the record. Only an explicit pki.trustVersion above the recorded version publishes such a bundle again. pki.trustVersion is now a floor, not something you must bump: the published version is never below it. Approver-set signatures no longer name a version: since c6214dc they sign the set's own epoch (pki.approvers.epoch), so they stay valid on every later version.

better-compute pki rotate-command-root --stage add|switch|drop runs one step on the bundle and writes it back (sealed when BETTER_COMPUTE_SECRET is set), printing the roots and the bundle's trust version (with the content rule above, the version the deployment publishes may be higher, never lower):

better-compute pki rotate-command-root --stage add     # a new root beside the current one; trust version up
# roll the bundle out to every process; daemons get the new version in Welcome.trust
better-compute pki rotate-command-root --stage switch  # workers sign under the new root; the old one stays trusted
# at least 24 hours later (worker keys live at most 24 h)
better-compute pki rotate-command-root --stage drop    # the old root is no longer trusted; version up again

A step out of order is INVALID_STATE, and doctor reminds you of a rotation left half done (pki.rotation). So is a run with BETTER_COMPUTE_PKI_ALLOW_UNAUTHENTICATED=1: the command writes the bundle back, and only pki reseal vouches for an unauthenticated one.

Every daemon reports the trust bundle version it holds: Hello.trust_version at session start, and StatusReport.trust_version again once it applies a newer bundle from Welcome.trust mid-session, so the acknowledgment does not wait for a reconnect. bcd and the FakeDaemon both send it. The gateway records it as node_sessions.trust_version (core/0023_session_trust_version; a status report only ever raises it). The --stage switch step reads that column and refuses with INVALID_STATE while any connected node is behind, naming a few (1 of 2 connected node(s) have not applied it (node_old at 1)), since those nodes would reject every command signed under the new root. Since the published version follows the content, the step gates on the version the deployment published for this bundle, not the bundle's own trustVersion: it recomputes the digest this bundle and configuration publish and refuses (Not switching: the deployment still publishes trust version N without the new root) until every process runs the --stage add bundle, then waits for daemons to hold that published version (03bdbc9). Connected means a session on a gateway, not ended, seen within the last 2 minutes; a node that connects later gets the current bundle in Welcome before any command. The check needs the deployment's configuration or BETTER_COMPUTE_DATABASE_URL and refuses without either; --force skips it. While a rotation is in progress, the pki.rotation finding of doctor and the support bundle says how many connected nodes already have the new version. addTrustRoot, trustVersionAcknowledged and removeTrustRoot in better-compute/pki implement the same gates for your own tooling.

In production the design keeps this root in KMS or Vault transit; the bundle form is for development and small deployments.

The destructive-batch quorum (v1.0)

Signed intent stops a gateway from forging commands, but not a compromised api or worker, which holds a worker key. PLAN.md §13's answer for fleet-wide destructive work (reimage, wipe, reboot of many machines) is a quorum checked on the machine: destructive commands carry a batch_id in the worker-signed body, and run only once k distinct operator approver keys, pinned in the trust bundle and held off the control plane, have signed a BatchApproval over the Merkle root of the batch's scope. Each command ships with its inclusion proof, so the executing agent checks membership itself and neither the control plane nor a gateway can widen the batch.

The scope. A batch's scope is a tree with one leaf per distinct (subject, kind, payload): H(0x00 ‖ lp(batch_id) ‖ lp(subject) ‖ lp(kind) ‖ lp(payload_sha256)), leaves ordered by hash, inner nodes H(0x01 ‖ left ‖ right) and an odd node promoted (RFC 6962 style; golden vectors in packages/protocol/golden/batch-scope.json). payload_sha256 binds a leaf to exactly one operation, or is left empty to allow any payload of that kind for that subject; the agent accepts a proof from either the bound or the unbound leaf (77cdf4f; command_batch_scope and command_batch_members carry it, migration quorum/0002_payload_digest). A leaf may also carry its payload itself, { typeUrl, value } (at most 64 KiB a leaf, and 8 MiB for all of a batch's leaves): its digest is then derived from those bytes (a payloadSha256 given as well must match), and the payload is stored (migration quorum/0007_leaf_payload) so approvers can decode the image, firmware or power action they sign, not only a hex digest (2ca0ba4). The typeUrl must be type.googleapis.com/ plus the kind's own payload type from the registry (fc2df7d, review 8), so a creator cannot label bytes as a harmless message.

Every subject must name a machine of the batch's own tenant (fc2df7d): the node itself, the site agent in bmc:<agent>/<address>, or the target in node:<id>. A subject naming another tenant's node is NOT_FOUND and a malformed one INVALID_INPUT, since approvers sign under the batch's tenant while the daemon checks only the leaf: otherwise a compromised worker could spend tenant A's approval on tenant B's machines. The subject is what the command acts on, computed by the executing agent from the signed body:

Without the BMC subject, "power-cycle 412 machines" would collapse into one (site agent, bmc.power) leaf, and approving it would let a compromised control plane power any machine behind that agent. BMC addresses are site-local, so the subject names the site agent too: an approval for a BMC at site A never admits a command at site B.

The wire. CommandBody.batch_id is inside the worker signature; Command.batch_approval (batch_id, scope_root, scope_size, not_after, signatures) and Command.batch_proof (index, siblings) travel outside it, since approvers sign the approval themselves. The TrustBundle gains approver_keys (kid, spki_der), approver_threshold, quorum_tiers (max_scope_size, threshold), unbatched_destructive_per_hour and approver_set_signatures (see changing the approver set), and CommandRejectReason gains BATCH_UNAPPROVED and UNBATCHED_LIMIT. The canonical quorum bytes approvers sign for a set change (approverSetInput: a u32 key count, then each key's length-prefixed kid and SPKI in kid order, the u32 threshold, the u32 tier count and each tier's pair, the u32 unbatched cap; all big-endian) have a golden vector beside batch-scope.json, packages/protocol/golden/approver-set.json, with the set digest and the signing input for an epoch, so bcd pins its encoding to the same bytes (26ab0fc). approver_set_epoch (field 8, c6214dc) numbers the quorum fields separately from the bundle's version. Approvals are signed under the purpose batch-approval, and the signed parts end with the deployment's trust domain, which every verifier takes from its own configuration: an approval signed for staging never admits a command in production. The server therefore needs pki.join (the trust domain) to accept approvals, and refuses the quorum with INVALID_CONFIG without it.

The primitives in better-compute/pki (packages/pki/src/quorum.ts, and approver-set.ts for set changes):

TrustBundleInput (for signJoinResponse and signTrustBundle) takes approverSetSignatures ({ kid, sig }) beside the quorum fields (6a7f1b7).

Publishing the keys. pki.approvers puts the quorum in the join-key-signed trust bundle (JoinResponse.trust and Welcome.trust), so a gateway can relay it but not change it. The bundle's version follows the change by itself (see the version follows the content). The first set needs nothing more, since a daemon without approver keys takes it as it comes (it only adds restrictions); any later change needs the current approvers (see changing the approver set):

export default defineConfig(() => ({
  pki: {
    // nodeCa and join as above
    commandSigning: bundle.commandSigning, // required with approvers
    approvers: {
      keys: [aliceSpkiDer, bobSpkiDer, carolSpkiDer], // P-256 SPKI DER, distinct
      threshold: 2, // 1 to the number of keys
      tiers: [{ maxScopeSize: 5, threshold: 1 }], // small batches need fewer approvals
      unbatchedDestructivePerHour: 0, // the default: unbatched destructive commands are break-glass only
    },
  },
}));

The options are validated at start (INVALID_CONFIG): commandSigning must be set, keys must be distinct P-256 public keys, thresholds whole numbers from 1 to the key count, tiers' maxScopeSize at least 1. Tiers must ascend strictly by maxScopeSize, their thresholds must never decrease, and none may exceed threshold (which applies to batches above every tier), so padding a scope with extra leaves never makes it cheaper to approve (d566cbe). A daemon cannot count fleet-wide, so per-daemon allowances for unbatched commands would let a compromised worker hit every node once; fleet-wide limits come from the approvers instead, for example a tier whose single signature comes from a rate-limited auto-approver service outside the api and workers.

The examples read the keys from BETTER_COMPUTE_APPROVER_KEYS (the .pub files of batch keygen, comma-separated), with BETTER_COMPUTE_APPROVER_THRESHOLD (required with keys, 1 to their count) and BETTER_COMPUTE_UNBATCHED_PER_HOUR (default 0), and refuse keys without a command-signing root in the bundle.

Batches on the server. The built-in quorum module, always hosted, owns migration quorum/0001_batches (command_batches, command_batch_scope, command_batch_members, command_batch_approvals). createCommandBatch(tx, { tenantId, scope, reason, notAfterMs? }, { now, quorum }) fixes a batch at creation: its root, size (at most 50,000 leaves) and every leaf's encoded BatchProof are stored, it ends after 24 hours by default, and it needs requiredApprovals for its size (0 without approver keys). A module reaches it through the command queue: ctx.commands.createBatch(tx, { tenantId, scope, reason, notAfterMs?, createdBy? }) stores a batch with the trust bundle's quorum and answers { batchId, held, required, size }, and ctx.commands.enqueue(tx, { ..., batchId }) puts a command into it. The batch must belong to the node's owner and be pending or approved and unexpired, and the command's (subject, kind) (pki's batchSubject) must be in its scope (INVALID_INPUT otherwise); batch_id goes into the signed body, a command_batch_members row is written, and while a quorum that needs approvals has not approved the batch the command is held (not_before infinity, no poke). The commands API shows it (a299e96): a batched command's CommandInfo carries its batchId, and a held one awaitingApproval: true without notBefore, until the batch is approved. enqueue records which leaf the command uses: the one bound to its payload's SHA-256 when the scope has it, else the unbound leaf. Without approver keys a batch is recorded for audit and its commands go out at once.

With approver keys and no break-glass allowance (unbatchedDestructivePerHour 0), enqueue refuses a destructive command without a batchId up front, since every daemon would refuse it UNBATCHED_LIMIT: INVALID_STATE with details: { reason: 'batch-required', kind }, naming commands.createBatch and the break-glass option (508728e). Destructive means the registry's CommandKindSpec.destructive, so plugin kinds count, and node.reboot is one. It is an early warning only; the daemon stays the judge. With an allowance, or without approver keys, unbatched destructive commands are enqueued as before.

The gateway sends a batched command only once it can prove it (40be696): when the batch needs approvals, dispatch selects a member only after the batch is approved, joining the member to its scope leaf and the batch in SQL (so held rows never crowd the dispatch limit), and sends it with Command.batch_approval and the member's own leaf's batch_proof, decoded from the stored bytes. A Hello replay attaches them again.

Who creates batches

With approver keys set, every built-in producer of destructive commands puts them in a batch before it sends anything, so none of them trips batch-required:

metalBatchScope(tx, registry, nodeIds, kinds) builds metal's scopes: the destructive kinds only, one leaf per node, and for a bmc.* kind one leaf per site agent of the node's site, since a site agent failover reroutes the command. The kinds come from profileCommandKinds(profile, steps) in provision, which relies on each control-plane step declaring what it sends: a plugin step that relays commands (Talos's httpViaSite) must declare sends: ['site.httpProxy'], or its commands fall outside the workflow's batch and are refused at enqueue (INVALID_INPUT, or batch-required when the workflow has no batch at all; see steps and profiles). Metal's leaves leave the payload unbound: the workflow builds its payloads later (a BMC credential rotated mid-workflow, user data rendered by an earlier step).

Over the API, a caller creates a batch with commandBatches.create and runs commands in it with commands.run({ ..., batchId }) (46afe75). create takes tenantId, a reason shown to approvers (at most 500 characters), the scope leaves ({ subject, kind, payloadSha256?, payload? }, payloadSha256 as hex, payload as { typeUrl, value } with value in base64; 1 to 50,000 of them), notAfterSeconds (60 s to 72 h, default 24 h) and an idempotencyKey. The batch id derives from the tenant, the caller's identity and the key, and migration quorum/0005_batch_request stores the request's digest, so a retry returns the same batch and approvers never sign two, while the same key reused for another request is CONFLICT. A new batch publishes command.batch.created. commands.run then checks the command's leaf against the scope and holds the command until approval, as enqueue does; with approver keys configured, even node.reboot needs such a batch. See running commands.

The commandBatches group (phase v1.0), decided on the tenant:

An agent's proposal reads as the agent's. When the caller of commandBatches.create is an agent acting through a delegation (MCP's propose_command_batch, or a delegated API call), the batch records the agent and the delegation beside the person in createdBy (command_batches.created_by_agent and delegation_id, migration quorum/0008_batch_agent; f3c5c4e, security review 14 M3), since its reason is then the agent's text, not the person's. batch approve notes "Proposed by agent X for Y (delegation D): the reason is the agent's own text" before the approver signs, batch show answers the same words as proposedBy, and the console's Batches page shows them as a warning line (bd80209); server strings pass through the CLI's printable filter first.

The IAM decision comes on top of the signatures, never instead of them. A pending batch past its end reads as expired, and the quorum.batches reconciler records that every minute and cancels its held commands; a deleted tenant's pending batches are cancelled.

The approvers' side. The better-compute batch commands (b3fd712, 6a7f1b7; flags in the CLI reference) never trust the control plane's word for what a batch holds:

batch approve refuses a batch that is not pending, pages the scope (up to 5,000 leaves a page) and recomputes the root from the listed (subject, kind, payload digest) leaves. Leaves that do not give the root and size the server asks approvers to sign, or more leaves than the size, are SCOPE_MISMATCH ("the control plane answered two different batches"); a page that does not advance, or an answer without a valid root and size, is INVALID_RESPONSE. It then shows the reason (and, for an agent's proposal, who proposed it for whom), every subject by kind (at most 12 per kind, then a count), and the approvals so far with the window's end, and signs the batch id, root, size, expiry and the deployment's trust domain. The review is enforced, not only shown:

• Without --yes it stops after the summary and prints the root; --root HEX approves only if the batch still has the root you reviewed (SCOPE_CHANGED otherwise).
• --expect FILE (subjects one per line, a JSON array of subjects, or { subject, kind } commands) approves only a batch that holds exactly those; SCOPE_UNEXPECTED names what is missing or not expected.
• Leaves that admit any payload are marked "any payload" and refused as UNBOUND_SCOPE, naming them, unless --allow-unbound.
• Bound leaves' operations are shown decoded, grouped by the payload's digest (each line shows its first 12 hex digits), with long strings shortened in the middle so every field shows; --operations-out FILE writes every distinct operation in full with its subjects (batch sign and batch approve; b118616). A payload with fields this build cannot decode, or whose re-encoding does not give the bytes back, is refused (payload-unknown-fields), and a kind whose payload type the CLI does not know is opaque and needs --allow-opaque; it is never decoded as the type the server labels it.
• A window that has closed, or that stays open longer than --max-hours (default 72, at most 720), is refused INVALID_STATE, since an approval admits its commands until then.
• An approver key readable by others is refused (POSIX; INVALID_STATE), as ssh does; an encrypted one needs BETTER_COMPUTE_APPROVER_PASSPHRASE (MISSING_ENV). A key that already approved the batch is reported and nothing is signed.

Once the batch holds the required approvals it is released and its commands run.

On the machine. bcd and the FakeDaemon (intent.quorum, or the quorum of its signed trust bundle) enforce it the same way. With approver keys in the trust bundle, only kinds the daemon's own table marks destructive are checked: one with a batch_id runs only with a batch_approval of enough distinct approver signatures (the tier for its scope size, else the threshold, never fewer than one; a kid must be the SHA-256 of its key; duplicates count once), not past its not_after in server time, and a batch_proof that proves the command's own (subject, kind) leaf, the subject taken from the signed body; else it is BATCH_UNAPPROVED. Each leaf runs at most once, since an approval for N machines is N operations, not N kinds of operation to repeat. Both daemons key a used leaf by the hash of the leaf the proof matched, which holds the batch id, so another approval of the same batch id over a different tree cannot move a used leaf to a fresh index (the FakeDaemon keyed it by batch id and leaf index until c589c5d): another command on a used leaf is BATCH_UNAPPROVED (the FakeDaemon's reason is leaf-used), while a redelivery or re-execution of the same command keeps its leaf. The FakeDaemon enforces this (d566cbe), and so does bcd, which keeps the leaves that ran, with their command, in data_dir/batch-leaves.json until the approval's not_after (8b0f50c). One without a batch_id counts against a rolling-hour allowance (unbatched_destructive_per_hour, default 0), beyond which it is UNBATCHED_LIMIT; redeliveries do not count twice, and bcd keeps the count in data_dir/unbatched-destructive.json so a restart does not reset it. The verdicts apply after the signature check. bcd keeps the whole trust bundle in identity.json.

Both daemons also hold these lines (3ff8131 and 8b0f50c in bcd, 6a7f1b7 in the FakeDaemon):

• A quorum change needs the current approvers. Once a daemon holds approver keys, it refuses a change of any quorum field (keys, threshold, tiers or the unbatched cap; removal included) unless the current threshold of distinct current approvers signed the new fields, since the JoinService key alone could otherwise drop the approvers or swap in its own. A current threshold of 0 admits no change. Both apply one epoch rule (c6214dc, 5fe7cbf, e171f0d): a refused set keeps the held quorum while the rest of the bundle (roots, version) applies; the FakeDaemon logs trust.quorum-refused with the reason. A daemon without approver keys takes the first set as it comes, or only its pinned set, and its epoch never goes back. Both report the set they hold in Hello and StatusReport (approver_set_epoch, approver_set_sha256).
• The unbatched window counts on the local clock, so a Welcome that moves server time forward cannot age the count out. Approvals still expire in server time.
• Nothing is used before admission. A batch leaf or an unbatched slot is recorded only once the command is accepted and checked again at admission, so a forged, re-targeted or expired delivery carrying a public approval and proof cannot burn a leaf or the cap (the FakeDaemon's claimQuorum, review 6 L1); bcd also checks a deferred command's approval again when it is finally admitted.

bcd alone refuses destructive commands when its stored bundle cannot be read, rather than switching the quorum off. bcd also treats a site.httpProxy other than a GET as a write (review 9): under a quorum it runs only under a leaf bound to its exact payload, never an unbound one, and never unbatched, whatever the hourly allowance; readiness polls, which are GETs, are unaffected. Both count bmc.boot and bmc.rotate as destructive, as metal's kind table does, so they are signed and batched like power actions. See signed commands.

Changing the approver set

The trust bundle is signed by the JoinService key, which the control plane holds, so on its own it must not be what changes the quorum. Adding or removing an approver, changing a threshold, a tier or the unbatched cap, or removing the quorum altogether therefore needs the current approvers' signatures over the new set (77cdf4f, 6a7f1b7, edbd63f). Since c6214dc (review 7 M1) they sign the set's own epoch, not the trust bundle version: TrustBundle.approver_set_epoch numbers the quorum fields, goes up with every change of the set, and stays put when roots rotate. approver_set_signatures are purpose approver-set over [uint64_be(epoch), SHA-256(approverSetInput), trust_domain]. Signing the version instead meant signatures were published only while that version was current, so a daemon offline during a set change met the next bundle (a root rotation, say) with the new set and no valid signatures, refused the whole bundle for good, and kept trusting rotated-out roots. The trust domain is the verifier's own (4835f4f, review 7): a staging set's published signatures would otherwise be replayable into production when the two share approver keys.

One rule governs every change (5fe7cbf and e171f0d, review 10 M3; the rule in common.proto): a daemon holding set H at epoch e takes (N, e') only when e' > e and at least H's threshold of H's keys signed (e', N, its own trust domain). That holds for N == H too: an unchanged set moves to a higher epoch only with its own signatures, and is otherwise kept silently at e. So a daemon that held S@1 and missed S -> T@2 and T -> S@3 cannot later be moved to T@2 by replaying that epoch's genuine signatures. The same set at the same epoch needs nothing; anything else is stale-epoch or approver-set-unsigned. approver_set_signatures may mix the previous set's signatures (the change) with the new set's own (which move daemons that already hold the new set to the new epoch); only held keys count. A daemon without keys takes the first set as it comes, or only its pinned one (below), never at or below an epoch it held, and keeps its epoch through a signed removal. When the quorum part is refused, the daemon keeps its quorum and still applies the rest of the bundle (roots, version). A daemon that missed two set changes whose keys do not overlap cannot verify the newer set with the keys it holds and needs re-provisioning (or re-pinning). The first set needs no signatures. On a fleet whose daemons already hold approver keys:

better-compute batch sign-approvers --key alice.pem --epoch 2 \
  --approvers alice.pem.pub,bob.pem.pub,dave.pem.pub --threshold 2 --tiers 5:1 --unbatched 0

pki: {
  commandSigning: bundle.commandSigning,
  approvers: {
    keys: [aliceSpkiDer, bobSpkiDer, daveSpkiDer],
    threshold: 2,
    tiers: [{ maxScopeSize: 5, threshold: 1 }],
    epoch: 2, // default 1; required with setSignatures: the epoch they sign
    setSignatures: [aliceSig, bobSig], // the { kid, sig } of sign-approvers (sig base64url or bytes)
  },
},

A quorum change without valid signatures for a higher epoch is still published, since daemons without approver keys take it, but each process logs a warning naming batch sign-approvers --epoch <published + 1>, and daemons that hold keys keep their old quorum until a signed set reaches them. To fix it, have the current approvers sign that epoch and set pki.approvers.epoch and setSignatures. better-compute doctor verifies setSignatures as a daemon does (the published set's threshold, the epoch, the deployment's trust domain), so signatures that do not verify are pki.quorum-unsigned with the reason (b118616).

Nodes that hold another set. A daemon can take a bundle's version while refusing its quorum part, so the trust version alone does not show which approvers a node enforces. The gateway records the set each daemon reports holding in node_sessions.approver_set_epoch and approver_set_sha256 (migration core/0026_session_approver_set, fa39201):

• Hello writes the reported set as it comes.
• A StatusReport with trust_version above 0 (the acknowledgement of a newer bundle) writes it only when that version is at least the one recorded, so a late, older report never pairs an old set with a newer version.
• A daemon that holds no approver keys (an empty digest) is recorded as NULL in both columns.

better-compute doctor and pki trust-version read those columns for the connected nodes (a session on a gateway, not ended, seen within the last 2 minutes) and compare them with the published set (6612496, 08eb62e). While the published bundle has approver keys, a node lags when its digest is NULL (it enforces no quorum at all, the worst lag), differs from the published set's, or its epoch differs. Without published keys, any node that still holds a set lags. Doctor reports pki.quorum-lag (a warning, only when a published approver set exists) as "N of M connected node(s) enforce another approver set than the published one (epoch E)", followed by up to five nodes with the epoch each holds (0 for a node without keys). pki trust-version prints the count as approverSetBehind and those nodes as approverSetSample (nodeId, epoch). Both leave the check out before core/0026 has run. The fix is to check the set's signatures, or to re-pin nodes that missed several changes.

The examples read the signatures from BETTER_COMPUTE_APPROVER_SET_SIGNATURES: the JSON files batch sign-approvers printed, comma-separated. They must all sign one epoch and exactly the quorum BETTER_COMPUTE_APPROVER_KEYS, BETTER_COMPUTE_APPROVER_THRESHOLD and BETTER_COMPUTE_UNBATCHED_PER_HOUR configure (the examples set no tiers), or the example refuses to start; that epoch becomes pki.approvers.epoch. BETTER_COMPUTE_TRUST_VERSION sets the trust bundle version's floor, independently.

Pinning the first set

A daemon without approver keys (every new or reimaged node) takes the first set its join-key-signed bundle carries, so a compromised api could hand it an empty set or its own key at k = 1, after which the operators' real set would need the attacker's signature. The first set can be pinned outside the api (9b7a199, review 7 M2): better-compute pki trust-version prints approverSetSha256 (SHA-256 of approverSetInput), computed from this configuration's approver keys rather than from the database the api writes, with publishedApproverSetSha256 beside it while the two differ (b118616, review 10 M4), and daemon install-script --approver-set-sha256 HEX writes it into the node's config.toml as approver_set_sha256 (an upgrade run without the flag keeps an existing pin, b6d401f). A daemon with that pin and no keys takes only a matching set (pin-mismatch otherwise, while the rest of the bundle still applies) and refuses destructive commands BATCH_UNAPPROVED (the FakeDaemon's reason is awaiting-pinned-set) until it holds one, so it fails closed. bcd (daemon/internal/pki/batch.go, agent/quorum.go) and the FakeDaemon (intent.approverSetSha256) both implement it.

Turning the quorum on

PLAN.md §13 records what happens when a deployment upgraded from v0.5 sets pki.approvers for the first time, and tests/sim/upgrade.test.ts proves it (80195df): no allocation moves, no VM reboots, and reservations stay active.

Exec tickets and the SSH CA

The exec ticket key signs ES256 JWTs (typ: bc-exec+jwt) that live at most 60 s, name one node as audience (aud: node:<id>), and carry a single-use jti. tunnels.open issues them after Better IAM allowed the call; the gateway and then the daemon verify each ticket against the JWKS in the node's NodeSpec. Every api replica and the gateway derive the same kid from the key, and rotate() keeps the previous key listed.

The SSH CA issues OpenSSH user certificates for break-glass access, valid at most 1 h and never beyond the requesting session, with the principal <account>@<nodeId> so a certificate opens only the node it names. Every issuance is a row in ssh_certificates; nodes.revokeSshCert adds the serial to the node's key revocation list in its next NodeSpec. See SSH and tunnels.

The sandbox SSH CA

The bundle's sandboxSshCa (v0.5) is a second OpenSSH CA, used only by sandboxes.exec. It signs user certificates of 60 seconds to 15 minutes, never past the caller's session or the sandbox's expiry, for the one principal <account>@<instanceId> (for a sandbox restored from a memory snapshot, the instance the memory was captured on, since the resumed guest keeps that principal). It is kept apart from the break-glass CA so neither certificate opens what the other is for: nodes never trust the sandbox CA, and sandbox guests never trust the break-glass CA.

• Signing. Pass it as tunnels.sandboxSsh (tunnelsOptions(bundle, …) does). Without it, sandboxes.exec is UNSUPPORTED. The tunnels module refuses a sandboxSsh whose key is the break-glass ssh CA's (INVALID_CONFIG: "tunnels.sandboxSsh must be a different CA from tunnels.ssh"), so the split cannot be undone by configuration.
• Trust. Pass its public key as vps({ sandboxExec: { caPublicKey } }). vps writes it into every sandbox instance's user data as TrustedUserCAKeys, with an AuthorizedPrincipalsFile holding that instance's own principal. vps and the tunnels module each refuse a key that is not one line (an ed25519 or ECDSA P-256 key for vps), since a second line would be a second trusted CA; the bundle loader refuses one too.
• Older bundles. pki add-sandbox-ca adds it; pki show prints it as sandboxSshCaPublicKey.
• Revocation. Guests have no agent and receive no revocation list, so the short lifetime and tunnel re-authorization are the controls. See sandboxes.
• Host certificates. Every SshCertificateAuthority can also sign OpenSSH host certificates: issueHostCertificate({ publicKey, principals, keyId, validBefore }) returns { certificate, serial, validBefore } with no options or extensions, for principals that are plain names (letters, digits, ., _, -; never a list or a wildcard) and a lifetime of at most maxHostTtlSeconds (default and maximum 30 days). Wire the sandbox CA's issueHostCertificate into vps({ sandboxExec: { certifyHostKey } }) and sandbox guests present host keys it certified, which sandbox exec pins with @cert-authority instead of accepting them on first use; see how the guest trusts it.

The audit key

With the bundle's auditSigning key, auditModule({ signer }) seals audit_events into signed, hash-chained, per-tenant Merkle checkpoints every hour. better-compute audit verify --tenant ID checks a tenant's trail against the key's pin, taken from --pin, BETTER_COMPUTE_AUDIT_PIN or the bundle's public half, and exits AUDIT_TAMPERED on any problem. See audit.

Workload identity

Workload identity (v0.5, optional) gives hosted VPS instances short-lived OIDC tokens. Its key is not part of the bundle: createWorkloadIssuer({ issuer, defaultAudience, keys? }) from @better-compute/pki holds its own P-256 keys, and you pass the issuer as betterCompute({ workloadIdentity: { issuer } }).

• Tokens. ES256 JWTs with iss, sub = "t:<tenantId>:instance:<instanceId>", aud (the default audience unless the request names an allowed one), iat, nbf, exp, jti, tenant_id and instance_id, living 60 to 900 seconds. Extra claims may not shadow those.
• Publication. The issuer URL must be https on port 443, without query. better-compute serve answers GET and HEAD on <issuer path>/.well-known/openid-configuration and <issuer path>/.well-known/jwks.json, public, cached 5 minutes, with access-control-allow-origin: *; the JWKS is read per request, so an added key shows at once. The issuer URL has to reach that host.
• Rotation, in three steps so no token ever fails verification: addKey() publishes a new key while the old one still signs; once verifiers have re-read the JWKS, promote(kid) signs with it; retire(kid) drops the old key, refused until its last token has expired (15 minutes plus a minute after it stopped signing). exportKeys() returns the keys as PKCS#8 PEM with their state, for storage; handle them as secrets.

The host asks for a token over its session with a WorkloadTokenRequest signed by its node key (purpose workload-token), and the answer is sealed to a one-time key (JWE typ: bcworkload+jwe, binding the request id, the instance and the expiry). createWorkloadTokenRequest and openWorkloadTokenResponse are the host's half, openWorkloadTokenRequest, sealWorkloadTokenResponse and refuseWorkloadTokenRequest the broker's. bcd carries the host's half in Go (NewWorkloadTokenRequest and OpenWorkloadTokenResponse in daemon/internal/pki, checked against fixtures the TypeScript side produces), and its metadata service (daemon/internal/imds) calls it for the instances of whichever runtime bcd runs; neither runtime of host builds (libvirt-kvm, cloud-hypervisor) has run on real hardware yet, so only fakehw builds' instances have been served end to end. What the broker checks is in VPS instances.

Rotation and expiry