better/compute
Nodes

Attestation

The attestors that admit machines (join-token, x509pop, provisioned, discovery, cloud-iid, tpm), what each proves, single-use evidence and two-round joins.

A join token proves that someone with access to your API wanted a machine to join. That is enough for a server you install by hand, but not for a machine that PXE-boots on its own, a cloud VM, or a server whose certificate expired over a long weekend. An answers the question for one kind of evidence: is this machine allowed to join, and as which node?

Each attestor is a @better-compute/core AttestorVerifier that runs inside JoinService's join transaction. JoinRequest.attestor names it (empty means join-token), JoinRequest.attestation carries its evidence, and a sealed token, when the request has one, is opened and handed over as evidence.token. Every refusal reaches the agent as the same ACCESS_DENIED; the reason goes only to the onRefusal hook and the security.join.refused event (see joining).

The attestors

AttestorPhaseRolesEvidenceProvesJoins as
join-tokenv0.1node, sitea sealed node or site tokenthe holder of a live token for this pool or sitea new node, or a reclaim with allowReclaim
x509popv0.2node, siteits last certificate chain and a proof by that keythe machine still holds the key of a recent certificatethe same node, in place
provisionedv0.2nodea sealed single-use provisioned token and a product UUID claimthe OS a deploy or provider bootstrap installedthe target node re-keyed, or a new node
discoveryv0.2ramdiska sealed single-use discovery token and a signed discovery assertionthe ramdisk the site's boot server just booteda ramdisk identity of at most 2 hours
cloud-iidv0.3node (configurable)a provider-signed instance identitythe cloud instance your provider account createdwhatever resolve answers
tpmv0.5node (configurable)TPM endorsement and attestation keys, then an activated credential, a quote and the AK's certification of the node keythe TPM holding that endorsement key, and (by default) that the same TPM holds the CSR's keythe node bound to that key

Only join-token is built into the server. You add the others, and one with id join-token replaces the built-in:

compute.ts
import { betterCompute, pkiJoinCrypto } from 'better-compute/server';
import {
  discoveryAttestor,
  provisionedAttestor,
  x509popAttestor,
} from 'better-compute/pki';
import {
  discoveryWorkflowNodeResolver,
  provisionedTargetResolver,
} from 'better-compute/provision';

// `bundle`: the PKI bundle `better-compute pki init` wrote, loaded with importPkiBundle(text, { secret }).
export const compute = betterCompute({
  // database, secret, baseURL, iam, modules, …
  pki: {
    nodeCa: bundle.nodeCa,
    join: pkiJoinCrypto({
      trustDomain: bundle.trustDomain,
      nodeCa: bundle.nodeCa,
      signingKey: bundle.joinSigning,
      encryptionKey: bundle.joinEncryption.privateKey,
    }),
  },
  attestors: [
    x509popAttestor(),
    provisionedAttestor({ resolveTarget: provisionedTargetResolver() }),
    discoveryAttestor({
      resolveWorkflowNode: discoveryWorkflowNodeResolver(),
      approval: 'unknown-machines',
    }),
  ],
});

The attestors from @better-compute/pki also take onRefusal(refusal), called with { attestor, reason, tokenId?, nodeId?, tenantId? } for your own audit or metrics.

join-token

The v0.1 attestor, described in joining. It locks the join_tokens row, checks the secret in constant time, revocation, expiry at compute_now(), the remaining uses and the role (node tokens admit node, site tokens admit site), and consumes one use. JoinService enforces the token's expect binding and reclaim flag for every token-based attestor, not only this one.

x509pop

A node's certificate lasts 24 hours. A machine that was powered off or cut off longer than that comes back with an expired certificate, and renewal refuses expired certificates. x509pop lets it re-join as itself: it presents its last certificate chain and a proof by the certificate's key, bound to its new CSR.

The evidence is UTF-8 JSON:

JoinRequest.attestation
{
  "v": 1,
  "nodeId": "node_…",
  "chain": ["<base64url DER leaf>", "<base64url DER issuer>"],
  "clientTime": 1790000000000,
  "sig": "<base64url r‖s>"
}

sig is an ES256 signature under purpose x509pop over the CSR's SHA-256, the node id and the client time, so neither a Renew proof nor an earlier x509pop proof can be replayed as one. The attestor refuses unless:

  • the leaf is byte for byte a node_certs row of that node, of its current identity generation, and not revoked;
  • the node is not retired, and the certificate's role is the requested one (node or site; a lessee's node-tenant certificate re-joins as node and JoinService restores the restriction);
  • the certificate expired at most graceMs ago (7 days by default);
  • clientTime is within windowMs (60 seconds) of database time, and the proof was not used before (replayCache, in memory by default, 10,000 entries).

The result names the node, so JoinService reclaims it in place: it raises the identity generation, revokes every older certificate and keeps the node's state. A revoked certificate never re-joins; neither does one from before nodes.revokeIdentity. Agents build the evidence with createX509popEvidence (or x509popProducer for FakeDaemon). bcd does not produce this evidence yet: it joins with join-token only and answers no attestor challenge, so a bcd node whose certificate expired offline cannot re-join this way yet.

provisioned

When Better Compute deploys an operating system, the deploy renders a single-use provisioned token into the machine's user data (the provisioning engine's userdata.render step). The installed system joins with it. The attestor requires:

  • a token of role provisioned with exactly one use left (not-single-use otherwise);
  • a product UUID in the machine's claims (no-product-uuid);
  • a binding to one machine: the token's expect.productUuid, or a target the host resolves (unbound).

resolveTarget(tx, token) answers with the node the token was minted for and its expected product UUID. provisionedTargetResolver() from @better-compute/provision finds the running workflow whose userdata.render step minted the token, and reads the product UUID from that node's newest inventory. With a target node, the join re-keys that node in place and keeps its provisioning state; a UUID that differs is product-uuid-mismatch. Without one, the machine joins as a new node, which is what a provider bootstrap such as Hetzner Robot's installimage does.

discovery

During commissioning, a bare-metal server PXE-boots a platform ramdisk before anything is installed on it. The ramdisk has no identity of its own, so two parties vouch for it: the iPXE script carries a single-use discovery token bound to the MAC, and the site's boot server signs a statement that it just served that MAC.

The evidence is the sealed token plus a serialized SignedDiscoveryAssertion (bettercompute.site.v1). The assertion names the site, the vouching site agent's node id, the boot lease epoch it held, the MAC, and optionally the IP, product UUID and workflow id, with issued_at and not_after. The attestor checks:

  • the token: role discovery, one use left, issued for the assertion's site, and its expect.mac (when set) equal to the asserted MAC;
  • the signer: a site node of that site, not retired, currently holding the site:<id>:boot lease at exactly the asserted epoch, so a fenced boot server can no longer vouch; the signature (purpose discovery-assertion) must verify against its unexpired certificate;
  • freshness: the assertion is valid until the earlier of its not_after and 15 minutes after issued_at (maxAgeMs);
  • the machine: its claimed MACs include the asserted MAC, and its product UUID matches when the assertion names one.

Each signed assertion admits one join (assertion-reused otherwise), and the token is spent. The resulting certificate is a ramdisk certificate of at most 2 hours (certTtlSeconds may shorten it) that allows only the commissioning command kinds, discoveryAllowedKinds by default: inventory.refresh, logs.collect, diag.*, disk.wipe, deploy.* and boot.kexec, which are exactly the registered kinds whose roles include ramdisk (tests/pki-attestors.test.ts checks the two agree). Pass allowedKinds to change the list.

OptionDefaultEffect
resolveWorkflowNode(tx, workflowId)noneA known machine re-commissions as its node (discoveryWorkflowNodeResolver() answers for running workflows). Without it every discovery creates a node.
approval'never''unknown-machines' defers machines no workflow names; 'always' defers every one.
hardwareKey(claims)noneThe trusted hardware key requests are keyed on; without one they key on site and MAC.
pendingTtlSeconds86,400How long a pending request waits for a decision.
retryAfterSeconds30When the ramdisk retries a deferred join.

Both halves in bcd

bcd produces this evidence on both sides. bcd site signs one assertion for every iPXE script its boot server renders, while it holds the boot lease: the site id, its own node id, the lease epoch, the MAC, the IP (from iPXE's query, else the connection), the product UUID (from iPXE, else the boot entry), the workflow, issued_at in server time, and a not_after of at most 15 minutes and never past the boot entry's, signed by its node key under purpose discovery-assertion. It fetches the machine's discovery token with a SecretRequest bound to the workflow and puts both on the ramdisk's kernel command line (bc.token, bc.assertion). bcd ramdisk reads them back, seals the token to the pinned JoinService key and sends Join with attestor discovery, role ramdisk, the serialized SignedDiscoveryAssertion as the attestation, and its hardware claims (MACs, product UUID). A deferral is retried with the same CSR. The Go tests check the assertion's fields and signature, and the TypeScript vectors pin the signing input, but no test yet joins a real bcd ramdisk through this attestor. See the boot server and the ramdisk.

cloud-iid

A cloud VM that your provider account created can prove it with the identity document its cloud signs, so it needs no token at all. cloudIidAttestor({ providers, resolve, roles? }) verifies the document with one of its providers, then asks resolve(tx, identity, { role, claims }) where the instance joins. resolve returns an attestation result (tenant, pool, site, and nodeId for a known machine) or undefined to refuse (unknown-instance), for example for an instance in an account you do not manage.

JoinRequest.attestation
{ "provider": "gcp", "document": "<identity token JWT>" }
ProviderVerifiesBound to the CSR
gcpInstanceIdentity({ audience, fetch or jwks, projects?, maxAgeMs? })A Compute Engine identity token with format=full, signed by Google (RS256 or ES256, JWKS cached an hour). projects limits the accepted project ids.Yes: the agent requests audience cloudIidAudience(audience, sha256(csr)), that is <audience>#<base64url hash>.
awsInstanceIdentity({ certificates, maxAgeMs?, accounts? })The EC2 identity document and its signature, checked with the region's AWS certificate, which you supply ('*' covers regions not listed). accounts limits the account ids.No: accepted only within maxAgeMs (30 minutes) of the instance's pendingTime.
oidcInstanceIdentity({ id, issuers, jwks, audience, identity })Any issuer that signs JWTs into a JWKS; identity(claims) maps the claims to account and instance.Yes, through the audience.

Identity tokens older than maxAgeMs (10 minutes by default for OIDC and GCP) are refused. Whatever the provider, each (provider, account, instance) admits exactly one join, ever (instance-reused): an AWS document is readable by any process on the VM, and even a bound token must not mint a second node. A known instance that lost its certificate re-joins with x509pop.

tpm

tpm (v0.5) ties a node to its TPM's endorsement key (EK), an identity that survives a reinstall or a disk swap. It follows the go-attestation enrollment flow as a two-round join.

  1. Challenge. The agent sends its EK (public area and manufacturer certificate) and an attestation key (AK). The attestor checks the EK certificate chains to one of ekRoots and that the AK has attestation-key attributes, then answers with a MakeCredential challenge plus an opaque state sealed under challengeKey (AES-256-GCM, bound to the CSR, at most 5 minutes). It writes no row, so any api replica can finish the join.
  2. Answer. Only the TPM holding that EK, with that AK loaded, can activate the credential. The agent returns the secret and a quote of its PCRs whose extraData is sha256(secret ‖ sha256(csr)). The attestor checks the secret, the quote signature, the nonce, that the PCR values match the quote's digest, and your pcrPolicy.
  3. Node key (cf24876). The same answer carries nodeKey { public, certifyInfo, signature }: a TPM2_Certify of the node key by the AK, qualified by sha256(csr). The attestor checks that the certification is signed by the AK it just proved beside the EK, that it is a TPM_ST_ATTEST_CERTIFY whose extraData is sha256(csr), that the certified name is the TPM name of public, that the key was made in the TPM and can never leave it (fixedTPM, fixedParent, sensitiveDataOrigin and sign set, decrypt clear), and that its SubjectPublicKeyInfo is the CSR's (JoinService passes its SHA-256 as csrSpkiSha256). A certification that fails a check is refused with node-key-malformed, node-key-signature, node-key-nonce, node-key-name, node-key-attributes or node-key-not-csr. With the default nodeKey: 'certified', an answer without one is refused too (node-key-not-certified); nodeKey: 'any' admits it, and the result then makes no claim about the key.

It then decides where the machine joins, keyed by the SHA-256 of the EK: the node that key is already bound to (a reclaim), else your resolve allow-list (for example a vendor's delivery manifest), else operator enrollment through join requests. JoinService binds the EK to the node (node_bindings): a later join proving it reclaims that node, and it can never admit another one (bound-elsewhere). When the node key was certified, the result carries nodeKey: 'tpm', and JoinService records it in the node.joined event and in the join's audit row (nodeKey: tpm, or unattested for every other join; see joining). Since d94f1f3 the result also carries nodeKeyUri, the AK's name as urn:bc:tpm-ak:<nameAlg>:<digest>, which JoinService writes into the certificate so that every renewal must be certified by the same AK (see renewing a TPM-held key).

OptionMeaning
challengeKey32 bytes shared by every api replica; seals the challenge state. Required.
ekRootsManufacturer EK CA certificates (PEM or DER).
allowUncertifiedEkAccept an EK without a certificate (swtpm, labs). Never in production.
pcrSelectionPCRs the quote must cover; default the SHA-256 bank, PCRs 0 to 7.
pcrPolicy(pcrs)Returns a refusal reason, or undefined to accept.
resolve(tx, identity, { role })Where an unbound EK joins, or undefined.
enrollment{ tenantId, siteId, poolId?, pendingTtlSeconds?, retryAfterSeconds? }: defer unknown EKs for approval.
challengeTtlMsChallenge lifetime, at most 5 minutes.
nodeKey'certified' (the default): admit a join only when the AK certifies the CSR's key as one this TPM made and never lets out. 'any': also admit a node key held elsewhere, such as a file.

nodeKey defaults to certified

Since cf24876 nodeKey: 'certified' is the default, so a tpm join whose agent keeps its node key outside the TPM is refused (node-key-not-certified), including a machine that joined with a file key before and joins again. Set nodeKey: 'any' to keep admitting those; their certificates then name no AK and renew without a TPM proof.

Renewing a TPM-held key

The join proves the first node key. Since d94f1f3 every renewal proves the next one the same way, so a certificate issued after a certified tpm join names, for its whole life and every renewal after it, a key that cannot leave that TPM.

  1. The statement. A certified join returns nodeKeyUri (AttestationResult.nodeKeyUri), and JoinService issues the certificate with two URI SANs: the SPIFFE ID first, then urn:bc:tpm-ak:<nameAlg>:<digest>, the AK's TPM name in hex (tpmAkUri; the name covers the AK's whole public area, so only that key matches it). A join without a certified key (any other attestor, or nodeKey: 'any' without a certification) gets no statement and renews as before. Since 0e42018 JoinService writes a statement only when renewals can prove it again (JoinCrypto.knowsKeyStatement; pkiJoinCrypto knows urn:bc:tpm-ak: alone): a join whose attestor returns any other nodeKeyUri is refused with node-key-statement (security.join.refused reason join:node-key-statement) rather than issued a certificate that could never renew.
  2. Where Renew reads it. From the presented leaf, which must already be byte for byte compute's own node_certs row, never from the CSR. A CSR that asks for no SAN at all still gets the statement back, and a CSR cannot remove it.
  3. The proof. A leaf with a urn:bc:tpm-ak: statement renews only with RenewRequest.key_attestation (field 5, TpmKeyAttestation): the AK's public area (ak_public), the new key's public area (key_public), and the AK's TPM2_Certify of that key (certify_info, signature) qualified by sha256(csr). verifyTpmKeyAttestation checks that the AK's TPM name is the one in the statement and that it has attestation-key attributes, then runs the same checks as the join's node-key proof: the AK's signature, the qualifying data, the certified name, fixedTPM, fixedParent, sensitiveDataOrigin and a signing key, and that the key's SPKI is the CSR's.
  4. No downgrade. A missing or failing attestation refuses the renewal with ACCESS_DENIED ("Renew refused"), and JoinService logs renew refused with the node id and the reason. Since 0e42018 it also publishes security.renew.refused to the node owner, naming the node (see refused renewals). The new certificate carries the statement forward, so the next renewal needs the AK again.
ReasonMeaning
key-attestation-missingThe leaf names an AK and the request has no key_attestation
key-attestation-akak_public is not the named AK (another TPM's), or is not an attestation key
node-key-malformed, node-key-signature, node-key-nonce, node-key-name, node-key-attributes, node-key-not-csrThe certification fails the same check as at join (a replayed certification for another CSR is node-key-nonce)
tpm-ak-uriThe statement in the leaf is malformed
key-statement-unknownThe leaf carries a urn:bc: statement this control plane does not know
key-statement-unsupportedThe JoinCrypto has no verifyKeyStatements (pkiJoinCrypto has one)

The node certificate's URI SANs are now exactly one SPIFFE ID followed only by urn:bc: statements (at most four URIs in all). The identity readers (nodeSpiffeIdOf in @better-compute/pki, used for mTLS identity, proofs of possession, join responses and the gateway's renew relay) and bcd's pki.NodeIdentity take the node id from the one SPIFFE ID and never from a statement; see PKI.

bcd's side. At each renewal renewOnce reads the current leaf. When it names an AK, bcd loads that AK alone (tpm.Certifier, a primary of the endorsement hierarchy, no EK), checks that its TPM name is the one the certificate names, derives the next node key in the same TPM connection, and has the AK certify it over sha256(csr) into key_attestation. A root bcd node does this on the TPM it opened; an unprivileged one under [helper] asks bcd helper (tpm.ak, tpm.nodekey, tpm.certify; 8870b8d), which holds the TPM connection, and checks that the helper certified with the AK it announced (see the node key in the TPM). There is no fallback: where a certificate without a statement would get a file key with a warning, this one fails the renewal instead and sends nothing.

At renewalWhat bcd does
The TPM cannot be openedrenewal: the certificate's node key is in the TPM, which is unavailable: ...
The TPM derives another AKrenewal: this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join (a1c1a46)
The TPM cannot make or certify the next keyrenewal: the next node key in the TPM: ... or renewal: the TPM's attestation key certifying the next key: ...

Each failure is logged as renewal failed with the attempt and expiresIn, and retried with backoff from 30 seconds to 10 minutes. Retries do not help a TPM that lost the AK or the key: the certificate (at most 24 hours) runs out, and the node goes offline.

Recovering a node whose TPM changed

A tpm-joined node recovers by joining again with its TPM. bcd join alone joins only when the old identity no longer loads; one that still loads answers already joined as <node> whatever state its certificate is in. Since a1c1a46 bcd join --reset --yes moves the old identity aside first, so the recovery command is the same whichever case you are in:

# as root on the machine
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm

--reset renames identity.json, node.key and a pending node.key.next to <name>.reset-<UTC time> in the data directory (nothing is deleted) and the join then runs as on a new machine; see joining again with --reset for its guards. The attestor keys the machine on the SHA-256 of its EK, so what happens depends on whether the EK survived:

  • The EK is the same (for example after TPM2_Clear, which resets the owner hierarchy's seed but not the endorsement seed the EK and AK come from). The node key is a primary of the owner hierarchy, so the same node.key record now derives another key, and bcd no longer loads its identity: "the node key does not match the node certificate: the TPM derives another key from node.key (was the TPM cleared?); join again: bcd join --reset --yes, then bcd join" (since a1c1a46 this hint is on the key mismatch, where a clear shows; the AK survives a clear). The join finds no usable identity, and the attestor reclaims the node that EK is bound to: the same node id, a new identity generation (every older certificate revoked), a new certified key and a fresh AK statement. Here a plain bcd join --attestor tpm would work too, since the identity no longer loads; --reset keeps the old files instead of overwriting them.
  • The EK is new (another TPM, a replaced board, a reset endorsement hierarchy). The renewal's AK check fails ("another TPM, or its endorsement seed changed"). The attestor does not know the EK, but the old node still exists: it holds the binding to the old EK and, usually, the machine's trusted hardware key, and a join whose hardware key another node holds is refused (reclaim), since a tpm join carries no allowReclaim. Remove the old node first, then join again:
    1. cordon the old node (better-compute nodes cordon, nodes.cordon) and wait until it holds no open allocation;
    2. delete it (better-compute nodes delete <node> --yes, 4049621; nodes.delete, which since 54cad9e accepts a cordoned adopted node): its certificates are revoked and its EK binding and trusted hardware key are freed (see removing an adopted host);
    3. on the machine, as root, run bcd join --reset --yes --attestor tpm. The machine joins wherever your resolve or enrollment sends it, as a new node (with neither, it is refused). It has a new node id; the --reset moved the old node's journal.db aside with the identity (b4fcc2e), so bcd node starts with a journal of its own.

bcd join --reset without --yes moves nothing: it prints what it would move and exits 2. The hints bcd prints spell the recovery as two runs, bcd join --reset --yes, then bcd join; one run with both does the same, because the reset happens before the join reads any credential. If the join after a reset fails, the identity stays moved aside and a later bcd join joins.

A node that joined with a token has no statement, so its renewal does not depend on the TPM's AK. Its node key, if in the TPM, still needs that TPM to load: after a clear it joins again with a token, and a token join of a known machine needs allowReclaim (see placeholder hardware and reclaims).

bcd with its TPM

tpmProducer(SoftwareTpm) is the agent side in TypeScript tests. bcd has its own, internal/tpm (f61b32d): a pure-Go TPM 2.0 client over the kernel resource manager (/dev/tpmrm0, Linux only) that derives the default RSA-2048 EK and a restricted P-256 AK and builds this attestor's { ek, ak } evidence and its round-2 answer (the ActivateCredential secret and a quote with extraData = sha256(secret || sha256(csr))), with a software TPM, tpmsim, for tests. Since 8545688 bcd join --attestor tpm uses it:

bcd join --config /etc/bcd/config.toml --attestor tpm
  • The TPM is the credential. No token is read or sent: BCD_JOIN_TOKEN is dropped from the environment, and --token-file with --attestor tpm is a usage error. Where the machine joins is the attestor's decision (the bound node, resolve or enrollment above).
  • Which TPM. A host build opens /dev/tpmrm0; a fakehw build keeps a software TPM in <data_dir>/fakehw-tpm.json, the same EK on every join of that machine.
  • The rounds. agent.Join sends the evidence, answers up to 3 challenge rounds for the same CSR, and waits out a pending approval (enrollment) before a fresh challenge round. A machine that already joined answers already joined as <node> without touching the TPM, unless --reset --yes moved its identity aside first (recovering a node whose TPM changed).
  • The node key, certified (cf24876). When node keys can use a TPM on this host (below), agent.Join makes the join's node key in the attestor's own TPM (tpm.Attestor.HoldNodeKey, through the NodeKeyHolder interface), and the round-2 answer carries the AK's TPM2_Certify of it over sha256(csr). If the TPM cannot make that key, the join stops with an error rather than falling back to a file. When node keys cannot use the TPM (a fakehw build without BCD_FAKEHW_TPM_KEYS=1), the key is a file, the answer carries no certification, and an attestor left at nodeKey: 'certified' refuses the join. bcd join runs as root and opens the TPM itself, [helper] or not; since 8870b8d it no longer checks whether the helper's user can open /dev/tpmrm0, because that user never does (below).
  • The install script. daemon install-script --attestor tpm (a006d9f) writes a script that reads no token, fails early without /dev/tpmrm0, and runs bcd join --config /etc/bcd/config.toml --attestor tpm < /dev/null before (re)starting the units, so the same script installs and upgrades.

The node key in the TPM

Independent of the attestor, bcd keeps its node key in the host's TPM when there is one (965c14d, 50e18f9, 0ba2a79). The node key signs the mTLS handshake, CSRs, renewal proofs, node statements (attested reports, command results, discovery assertions, secret and workload-token requests) and the WebSocket proof; in bcd it is a Go crypto.Signer, whatever holds it, and the wire does not change: the same P1363 signatures and the same kid (SHA-256 of the key's SPKI).

  • The key. tpm.NodeKeyTemplate is an unrestricted P-256 signing key (fixedTPM, fixedParent, sensitiveDataOrigin, userWithAuth, noDA, sign), a primary of the owner hierarchy derived from a 32-byte per-key unique. Nothing is stored in or evicted to the TPM: bcd keeps the unique and derives the same key again at every load. Each signature uses ECDSA with the digest's hash (SHA-256, -384 or -512, for TLS).
  • On disk. node.key (and node.key.next during a renewal) then holds only the record: a PEM block of type BCD TPM NODE KEY whose bytes are the unique, mode 0600. A key in the TPM does not load without that TPM.
  • When. Every new node key is made in the TPM when there is one: at bcd join and at every renewal's next key (under [helper], in the helper's TPM connection). A host build uses /dev/tpmrm0 when it exists; a fakehw build uses its software TPM only with BCD_FAKEHW_TPM_KEYS=1. When the TPM cannot hold the key, bcd logs a warning and makes a PKCS#8 file key instead, except at the renewal of a certificate that names an AK, which fails rather than leave the TPM (see renewing a TPM-held key). A node that joined with a file key moves into the TPM at its next renewal.
  • Under [helper] (8870b8d, security review 3). The unprivileged bcd node never opens the TPM. The kernel's resource manager filters no command, so whoever can open /dev/tpmrm0 can change the hierarchies' authorization, clear the TPM through its (empty) lockout authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys) and attestation. bcd join and bcd helper run as root and open the TPM themselves; an unprivileged bcd node reaches its node keys only through the helper's tpm.* operations, which name a key by its 32-byte unique, and checks every signature the helper returns against the key's public half. The helper builds every TPM command itself, so the node picks no template, handle, hierarchy or authorization (see node keys through the helper). Install scripts from 0ba2a79 until 8870b8d put the bcd user in the tss group instead; an upgrade run takes it out again (see running bcd node unprivileged).
  • What it protects. Such a key cannot be copied off the machine, but anyone with the owner hierarchy's authorization on that TPM (root on the node) can derive and use it: non-exportable, not root-proof.
  • Seeing it. bcd doctor has a node key check: ok "in the TPM", ok for a file key on a host with no TPM, and warn for a file key on a host with a TPM ("the next renewal moves it into the TPM (through bcd helper when bcd node runs unprivileged)"). Under [helper] its tpm check (e6c3f53) fails while the node's user is still in the tss group. bcd status prints node key in the TPM or node key a file on its identity line, and its JSON carries nodeKey (tpm or file).

What the control plane knows about it depends on how the key was made:

WhenProven to the control planeWhere it shows
A tpm join (cf24876)Yes: the AK certifies the key (CertifyNodeKey, TPM2_Certify over sha256(csr)) and tpmAttestor checks it, refusing the join under the default nodeKey: 'certified' when it is missingnodeKey: 'tpm' in node.joined and the node.joined or node.reclaimed audit row; urn:bc:tpm-ak: in the certificate (d94f1f3)
A join with any other attestorNo: bcd still makes the key in the TPM when it can, but sends no certificationnodeKey: 'unattested'; no statement in the certificate
A renewal of a certificate that names an AK (d94f1f3)Yes: the same AK certifies the next key (RenewRequest.key_attestation), and JoinService refuses the renewal without itthe statement, carried into the new certificate and its node_certs row
A renewal of any other certificateNo: the next key is made in the TPM when bcd can open one, but nothing proves itnothing; the certificate and node_certs do not record where the key lives

So for a certified tpm join the proof lasts: every certificate the node holds from then on names a key that its TPM made and certified. For every other node the control plane has only bcd's word (bcd status and bcd doctor report it locally).

What is tested, and with what TPM:

TestTPMWhat it proves
TestTSContract (daemon/internal/tpm)tpmsim fixture, with a manufacturer EK certificate in NV (b973e05)bcd's round-1 evidence is still byte for byte the fixture's, including the EK certificate bcd reads from NV index 0x01C00002 (tpm.EKCertIndex) and sends as ekCert, and the challenge tpmAttestor made for it activates and quotes through bcd's client (TypeScript to Go); since cf24876 the answer also carries the AK's certification of a node key held in the sim (answer.nodeKey)
tests/pki-attestors-tpm.test.tstpmProducer over a SoftwareTpm whose nodeKey it certifies as bcd doesunder nodeKey: 'certified', a certified CSR key is admitted with nodeKey: 'tpm'; no certification is node-key-not-certified, and a certified key other than the CSR's is node-key-not-csr; nodeKey: 'any' admits the uncertified answer without a nodeKey claim
tests/pki-attestors-tpm-bcd.test.tsthe same fixturetpmAttestor with ekRoots: [ek-root.der] and no allowUncertifiedEk verifies the EK certificate's chain, challenges bcd's evidence and admits bcd's answer, with its node key certified (nodeKey: 'tpm'), while a CSR key other than the certified one is node-key-not-csr; it refuses the EK under another root (ek-certificate-chain), and the answer for another CSR (challenge-for-another-csr) or after the challenge expired (challenge-expired) (Go to TypeScript)
tests/e2e/bcd-tpm.test.ts (BCD_E2E=1, WSL or Linux; db9bd46, 8a9af54)tpmsim in a fakehw bcd, with an EK certificate from a manufacturer root the test makes (BCD_FAKEHW_TPM_EK_CA) and the node key in that TPM (BCD_FAKEHW_TPM_KEYS=1)a real bcd join --attestor tpm with no token against the dev stack's JoinService, whose tpmAttestor trusts only that root (ekRoots, no allowUncertifiedEk): two rounds, resolve seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in the resolved pool and bound to sha256(EK) in node_bindings; node.key a BCD TPM NODE KEY record; the node key certified by the AK, so node.joined says nodeKey: 'tpm' for this join and for the rejoin below (cf24876); with its identity removed (a reinstall) the same TPM joins as the same node; bcd node on that identity, signing its session and statements in the TPM, is adopted; since d94f1f3 the certificate carries the AK's urn:bc:tpm-ak: statement, and bcd node renews (BCD_RENEW_AFTER=3s) with the AK certifying the next key, which stays a BCD TPM NODE KEY record while the new certificate keeps the statement
tests/server-join-tpm-renew.test.ts (d94f1f3)tpmProducer and softwareTpmCertify over SoftwareTpma certified join names the AK; a renewal without key_attestation is key-attestation-missing, another TPM's AK key-attestation-ak, a certification of another key node-key-not-csr, one for another CSR node-key-nonce; a good renewal keeps the statement although its CSR asks for no SAN, and the next renewal needs the AK again; an uncertified join (nodeKey: 'any') gets no statement and renews without one
internal/agent TestRenewCertifiedByTheAK (d94f1f3)tpmsima certificate naming the sim's AK renews with the next key made in that TPM and certified by it; with no TPM the renewal fails (unavailable), and with another TPM (another AK) it fails before any request, never falling back to a file key
internal/helper TestHelperTPMKeys (8870b8d, Linux)tpmsim behind a helper server on a test socketthe key and AK the node gets through tpm.* are the same as the TPM's own; signing and certifying; at most 8 keys loaded (least recently used unloaded) and a released key derived again on use; refusals of bad shapes, unknown verbs and a helper without a TPM

scripts/daemon-tpm-fixture.mjs regenerates the fixtures in daemon/internal/tpm/testdata after a deliberate change of either side, so a format change fails a test rather than a machine's join. Since b973e05 the contract runs the attestor as against a real TPM: ek-root.der is a test manufacturer root and ek-cert.der the EK certificate it issued for the sim's EK (made once by the Go test, and kept with sim.json when the fixtures are regenerated), and the attestor verifies that chain through ekRoots. Since 8a9af54 the dev-stack e2e does the same: a fakehw bcd given BCD_FAKEHW_TPM_EK_CA (a PEM file with the root's certificate and its PKCS#8 key) issues its software TPM an EK certificate in NV at each load, so allowUncertifiedEk is no longer used by any bcd test. Every test uses a software TPM: no test has run against a hardware TPM, a real manufacturer's EK certificate or a real /dev/tpmrm0.

Single-use evidence

Tokens are not the only thing that must not be used twice. cloud-iid and discovery record what they accepted in attestation_uses (attestor, subject, expires_at) inside the join transaction:

AttestorSubjectKept until
cloud-iidSHA-256 of provider, account and instanceforever
discoverySHA-256 of the signed assertion bodythe assertion's expiry

A second join with the same subject is refused. pruneAttestationUses(tx, { batch? }) from @better-compute/postgres deletes expired rows, 5,000 per call by default; see retention. Single-use tokens (provisioned, discovery) must have exactly one use left, so a token minted for several machines is never accepted as a machine credential.

Two-round joins

Some attestors cannot decide in one request: tpm must challenge the TPM, and discovery or tpm may wait for an operator. Such an attestor returns an AttestationDeferral instead of a result:

interface AttestationDeferral {
  deferred: 'approval' | 'challenge';
  requestId?: string; // approval: the join request
  retryAfterSeconds?: number; // approval: default 30
  challenge?: Uint8Array; // challenge: attestor-specific bytes
  notAfter?: Date; // challenge: answers after this are refused
}

JoinService then commits the transaction (so the attestor's own rows, such as a pending join request, persist), issues nothing and spends nothing, and answers with a signed JoinResponseBody whose deferral holds either JoinPending{ request_id, retry_after_seconds } or JoinChallenge{ attestor, payload, not_after }. The node id and chain are empty, and csr_sha256 still binds the answer to the request. The agent re-sends Join with the same CSR: after the wait, or with its answer to the challenge in attestation. Attestors therefore consume tokens and record uses only on the final round.

On the agent side, openJoinResponse reads a response as joined, pending or challenge, and verifyJoinResponse refuses a deferral. runAgentJoin in @better-compute/testing is the reference retry loop.

Requiring an attestor

By default a pool takes a machine whichever attestor admitted it. Since 1516728 a pool can require hardware attestation with policy.joinAttestors, the ids of the attestors that may admit a machine into it:

tpm-pool.ts
const pool = await client.pools.get({ tenantId: 'root', poolId: 'pool_01k…' });
await client.pools.update({
  tenantId: 'root',
  poolId: pool.id,
  // The policy is replaced as a whole, so keep the rest of it.
  policy: { ...pool.policy, joinAttestors: ['tpm'] }, // or ['tpm', 'x509pop'], or ['join-token', 'tpm'] to keep tokens
});
  • Where it is checked. JoinService checks it in the transaction that resolves the pool, before any reclaim or insert. A join or a reclaim admitted by an attestor that is not listed is refused with ACCESS_DENIED, reason pool-attestor. A token join is join-token, so a pool that lists only tpm refuses tokens, and a token reclaim of a TPM-bound machine (allowReclaim with the same trusted hardware key) is refused while its TPM still reclaims it.
  • Who sees the refusal. The attestor has already verified the machine, so the refusal is attributed to the pool's owner: security.join.refused with the attestor and reason join:pool-attestor, and the token or node id when known.
  • Validation. pools.create and pools.update refuse (INVALID_INPUT) a value that is not an array of 1 to 16 ids of attestors this control plane runs: join-token and every attestor registered in attestors or by a module or plugin.
  • Moving nodes. No API moves a node between pools. A node gets its pool at join, where the check runs, or at metal.enroll, whose machine gets its identity only through JoinService.

Tested by tests/server-join-pool-attestors.test.ts.

Choosing an attestor

SituationUse
A server you install by hand or with configuration managementjoin-token, with expect when the firmware reports placeholder identifiers
A node whose certificate expired while it was offlinex509pop
The OS a Better Compute deploy or a provider bootstrap installedprovisioned
A ramdisk during bare-metal commissioningdiscovery, with approval if unexpected machines should wait for a person
A VM your provider account createdcloud-iid
Hardware identity that must survive reinstalls, on TPM 2.0 machinestpm (v0.5), with bcd join --attestor tpm; add policy.joinAttestors: ['tpm'] to refuse every other way in

To write your own attestor, see attestors.

Was this page helpful?

Better Compute is created by Sean Filimon

Last updated

On this page