Attestation
The attestors that admit machines (join-token, x509pop, provisioned, discovery, cloud-iid, tpm), what each proves, single-use evidence and two-round joins.
A join token proves that someone with access to your API wanted a machine to join. That is enough for a server you install by hand, but not for a machine that PXE-boots on its own, a cloud VM, or a server whose certificate expired over a long weekend. An answers the question for one kind of evidence: is this machine allowed to join, and as which node?
Each attestor is a @better-compute/core AttestorVerifier that runs inside JoinService's join transaction.
JoinRequest.attestor names it (empty means join-token), JoinRequest.attestation carries its evidence, and a
sealed token, when the request has one, is opened and handed over as evidence.token. Every refusal reaches the
agent as the same ACCESS_DENIED; the reason goes only to the onRefusal hook and the security.join.refused
event (see joining).
The attestors
| Attestor | Phase | Roles | Evidence | Proves | Joins as |
|---|---|---|---|---|---|
join-token | v0.1 | node, site | a sealed node or site token | the holder of a live token for this pool or site | a new node, or a reclaim with allowReclaim |
x509pop | v0.2 | node, site | its last certificate chain and a proof by that key | the machine still holds the key of a recent certificate | the same node, in place |
provisioned | v0.2 | node | a sealed single-use provisioned token and a product UUID claim | the OS a deploy or provider bootstrap installed | the target node re-keyed, or a new node |
discovery | v0.2 | ramdisk | a sealed single-use discovery token and a signed discovery assertion | the ramdisk the site's boot server just booted | a ramdisk identity of at most 2 hours |
cloud-iid | v0.3 | node (configurable) | a provider-signed instance identity | the cloud instance your provider account created | whatever resolve answers |
tpm | v0.5 | node (configurable) | TPM endorsement and attestation keys, then an activated credential, a quote and the AK's certification of the node key | the TPM holding that endorsement key, and (by default) that the same TPM holds the CSR's key | the node bound to that key |
Only join-token is built into the server. You add the others, and one with id join-token replaces the
built-in:
import { betterCompute, pkiJoinCrypto } from 'better-compute/server';
import {
discoveryAttestor,
provisionedAttestor,
x509popAttestor,
} from 'better-compute/pki';
import {
discoveryWorkflowNodeResolver,
provisionedTargetResolver,
} from 'better-compute/provision';
// `bundle`: the PKI bundle `better-compute pki init` wrote, loaded with importPkiBundle(text, { secret }).
export const compute = betterCompute({
// database, secret, baseURL, iam, modules, …
pki: {
nodeCa: bundle.nodeCa,
join: pkiJoinCrypto({
trustDomain: bundle.trustDomain,
nodeCa: bundle.nodeCa,
signingKey: bundle.joinSigning,
encryptionKey: bundle.joinEncryption.privateKey,
}),
},
attestors: [
x509popAttestor(),
provisionedAttestor({ resolveTarget: provisionedTargetResolver() }),
discoveryAttestor({
resolveWorkflowNode: discoveryWorkflowNodeResolver(),
approval: 'unknown-machines',
}),
],
});The attestors from @better-compute/pki also take onRefusal(refusal), called with
{ attestor, reason, tokenId?, nodeId?, tenantId? } for your own audit or metrics.
join-token
The v0.1 attestor, described in joining. It locks the join_tokens row, checks
the secret in constant time, revocation, expiry at compute_now(), the remaining uses and the role (node
tokens admit node, site tokens admit site), and consumes one use. JoinService enforces the token's
expect binding and reclaim flag for every token-based attestor, not only this one.
x509pop
A node's certificate lasts 24 hours. A machine that was powered off or cut off longer than that comes back with
an expired certificate, and renewal refuses expired certificates. x509pop lets it re-join as itself: it
presents its last certificate chain and a proof by the certificate's key, bound to its new CSR.
The evidence is UTF-8 JSON:
{
"v": 1,
"nodeId": "node_…",
"chain": ["<base64url DER leaf>", "<base64url DER issuer>"],
"clientTime": 1790000000000,
"sig": "<base64url r‖s>"
}sig is an ES256 signature under purpose x509pop over the CSR's SHA-256, the node id and the client time, so
neither a Renew proof nor an earlier x509pop proof can be replayed as one. The attestor refuses unless:
- the leaf is byte for byte a
node_certsrow of that node, of its current identity generation, and not revoked; - the node is not
retired, and the certificate's role is the requested one (nodeorsite; a lessee'snode-tenantcertificate re-joins asnodeand JoinService restores the restriction); - the certificate expired at most
graceMsago (7 days by default); clientTimeis withinwindowMs(60 seconds) of database time, and the proof was not used before (replayCache, in memory by default, 10,000 entries).
The result names the node, so JoinService reclaims it in place: it raises the identity generation, revokes every
older certificate and keeps the node's state. A revoked certificate never re-joins; neither does one from before
nodes.revokeIdentity. Agents build the evidence with createX509popEvidence (or x509popProducer for
FakeDaemon). bcd does not produce this evidence yet: it joins with join-token only and answers no attestor
challenge, so a bcd node whose certificate expired offline cannot re-join this way yet.
provisioned
When Better Compute deploys an operating system, the deploy renders a single-use provisioned token into the
machine's user data (the provisioning engine's userdata.render step). The installed system joins with it.
The attestor requires:
- a token of role
provisionedwith exactly one use left (not-single-useotherwise); - a product UUID in the machine's claims (
no-product-uuid); - a binding to one machine: the token's
expect.productUuid, or a target the host resolves (unbound).
resolveTarget(tx, token) answers with the node the token was minted for and its expected product UUID.
provisionedTargetResolver() from @better-compute/provision finds the running workflow whose userdata.render
step minted the token, and reads the product UUID from that node's newest inventory. With a target node, the
join re-keys that node in place and keeps its provisioning state; a UUID that differs is
product-uuid-mismatch. Without one, the machine joins as a new node, which is what a provider bootstrap such as
Hetzner Robot's installimage does.
discovery
During commissioning, a bare-metal server PXE-boots a platform ramdisk before anything is installed on it. The
ramdisk has no identity of its own, so two parties vouch for it: the iPXE script carries a single-use
discovery token bound to the MAC, and the site's boot server signs a statement that it just served that MAC.
The evidence is the sealed token plus a serialized SignedDiscoveryAssertion (bettercompute.site.v1). The
assertion names the site, the vouching site agent's node id, the boot lease epoch it held, the MAC, and
optionally the IP, product UUID and workflow id, with issued_at and not_after. The attestor checks:
- the token: role
discovery, one use left, issued for the assertion's site, and itsexpect.mac(when set) equal to the asserted MAC; - the signer: a
sitenode of that site, not retired, currently holding thesite:<id>:bootlease at exactly the asserted epoch, so a fenced boot server can no longer vouch; the signature (purposediscovery-assertion) must verify against its unexpired certificate; - freshness: the assertion is valid until the earlier of its
not_afterand 15 minutes afterissued_at(maxAgeMs); - the machine: its claimed MACs include the asserted MAC, and its product UUID matches when the assertion names one.
Each signed assertion admits one join (assertion-reused otherwise), and the token is spent. The resulting
certificate is a ramdisk certificate of at most 2 hours (certTtlSeconds may shorten it) that allows only the
commissioning command kinds, discoveryAllowedKinds by default: inventory.refresh, logs.collect, diag.*,
disk.wipe, deploy.* and boot.kexec, which are exactly the registered kinds whose roles include ramdisk
(tests/pki-attestors.test.ts checks the two agree). Pass allowedKinds to change the list.
| Option | Default | Effect |
|---|---|---|
resolveWorkflowNode(tx, workflowId) | none | A known machine re-commissions as its node (discoveryWorkflowNodeResolver() answers for running workflows). Without it every discovery creates a node. |
approval | 'never' | 'unknown-machines' defers machines no workflow names; 'always' defers every one. |
hardwareKey(claims) | none | The trusted hardware key requests are keyed on; without one they key on site and MAC. |
pendingTtlSeconds | 86,400 | How long a pending request waits for a decision. |
retryAfterSeconds | 30 | When the ramdisk retries a deferred join. |
Both halves in bcd
bcd produces this evidence on both sides. bcd site signs one assertion for every iPXE script its boot server
renders, while it holds the boot lease: the site id, its own node id, the lease epoch, the MAC, the IP (from iPXE's
query, else the connection), the product UUID (from iPXE, else the boot entry), the workflow, issued_at in server
time, and a not_after of at most 15 minutes and never past the boot entry's, signed by its node key under purpose
discovery-assertion. It fetches the machine's discovery token with a SecretRequest bound to the workflow and puts
both on the ramdisk's kernel command line (bc.token, bc.assertion). bcd ramdisk reads them back, seals the
token to the pinned JoinService key and sends Join with attestor discovery, role ramdisk, the serialized
SignedDiscoveryAssertion as the attestation, and its hardware claims (MACs, product UUID). A deferral is retried
with the same CSR. The Go tests check the assertion's fields and signature, and the TypeScript vectors pin the
signing input, but no test yet joins a real bcd ramdisk through this attestor. See
the boot server and
the ramdisk.
cloud-iid
A cloud VM that your provider account created can prove it with the identity document its cloud signs, so it
needs no token at all. cloudIidAttestor({ providers, resolve, roles? }) verifies the document with one of its
providers, then asks resolve(tx, identity, { role, claims }) where the instance joins. resolve returns an
attestation result (tenant, pool, site, and nodeId for a known machine) or undefined to refuse
(unknown-instance), for example for an instance in an account you do not manage.
{ "provider": "gcp", "document": "<identity token JWT>" }| Provider | Verifies | Bound to the CSR |
|---|---|---|
gcpInstanceIdentity({ audience, fetch or jwks, projects?, maxAgeMs? }) | A Compute Engine identity token with format=full, signed by Google (RS256 or ES256, JWKS cached an hour). projects limits the accepted project ids. | Yes: the agent requests audience cloudIidAudience(audience, sha256(csr)), that is <audience>#<base64url hash>. |
awsInstanceIdentity({ certificates, maxAgeMs?, accounts? }) | The EC2 identity document and its signature, checked with the region's AWS certificate, which you supply ('*' covers regions not listed). accounts limits the account ids. | No: accepted only within maxAgeMs (30 minutes) of the instance's pendingTime. |
oidcInstanceIdentity({ id, issuers, jwks, audience, identity }) | Any issuer that signs JWTs into a JWKS; identity(claims) maps the claims to account and instance. | Yes, through the audience. |
Identity tokens older than maxAgeMs (10 minutes by default for OIDC and GCP) are refused. Whatever the
provider, each (provider, account, instance) admits exactly one join, ever (instance-reused): an AWS document
is readable by any process on the VM, and even a bound token must not mint a second node. A known instance that
lost its certificate re-joins with x509pop.
tpm
tpm (v0.5) ties a node to its TPM's endorsement key (EK), an identity that survives a reinstall or a disk swap.
It follows the go-attestation enrollment flow as a two-round join.
- Challenge. The agent sends its EK (public area and manufacturer certificate) and an attestation key (AK).
The attestor checks the EK certificate chains to one of
ekRootsand that the AK has attestation-key attributes, then answers with aMakeCredentialchallenge plus an opaque state sealed underchallengeKey(AES-256-GCM, bound to the CSR, at most 5 minutes). It writes no row, so anyapireplica can finish the join. - Answer. Only the TPM holding that EK, with that AK loaded, can activate the credential. The agent returns
the secret and a quote of its PCRs whose
extraDataissha256(secret ‖ sha256(csr)). The attestor checks the secret, the quote signature, the nonce, that the PCR values match the quote's digest, and yourpcrPolicy. - Node key (cf24876). The same answer carries
nodeKey { public, certifyInfo, signature }: aTPM2_Certifyof the node key by the AK, qualified bysha256(csr). The attestor checks that the certification is signed by the AK it just proved beside the EK, that it is aTPM_ST_ATTEST_CERTIFYwhoseextraDataissha256(csr), that the certified name is the TPM name ofpublic, that the key was made in the TPM and can never leave it (fixedTPM,fixedParent,sensitiveDataOriginandsignset,decryptclear), and that its SubjectPublicKeyInfo is the CSR's (JoinService passes its SHA-256 ascsrSpkiSha256). A certification that fails a check is refused withnode-key-malformed,node-key-signature,node-key-nonce,node-key-name,node-key-attributesornode-key-not-csr. With the defaultnodeKey: 'certified', an answer without one is refused too (node-key-not-certified);nodeKey: 'any'admits it, and the result then makes no claim about the key.
It then decides where the machine joins, keyed by the SHA-256 of the EK: the node that key is already bound to
(a reclaim), else your resolve allow-list (for example a vendor's delivery manifest), else operator
enrollment through join requests. JoinService binds the EK to the node
(node_bindings): a later join proving it reclaims that node, and it can never admit another one
(bound-elsewhere). When the node key was certified, the result carries nodeKey: 'tpm', and JoinService records
it in the node.joined event and in the join's audit row (nodeKey: tpm, or unattested for every other join;
see joining). Since d94f1f3 the result also carries nodeKeyUri, the AK's
name as urn:bc:tpm-ak:<nameAlg>:<digest>, which JoinService writes into the certificate so that every renewal must
be certified by the same AK (see renewing a TPM-held key).
| Option | Meaning |
|---|---|
challengeKey | 32 bytes shared by every api replica; seals the challenge state. Required. |
ekRoots | Manufacturer EK CA certificates (PEM or DER). |
allowUncertifiedEk | Accept an EK without a certificate (swtpm, labs). Never in production. |
pcrSelection | PCRs the quote must cover; default the SHA-256 bank, PCRs 0 to 7. |
pcrPolicy(pcrs) | Returns a refusal reason, or undefined to accept. |
resolve(tx, identity, { role }) | Where an unbound EK joins, or undefined. |
enrollment | { tenantId, siteId, poolId?, pendingTtlSeconds?, retryAfterSeconds? }: defer unknown EKs for approval. |
challengeTtlMs | Challenge lifetime, at most 5 minutes. |
nodeKey | 'certified' (the default): admit a join only when the AK certifies the CSR's key as one this TPM made and never lets out. 'any': also admit a node key held elsewhere, such as a file. |
Renewing a TPM-held key
The join proves the first node key. Since d94f1f3 every renewal proves the next one the same way, so a certificate
issued after a certified tpm join names, for its whole life and every renewal after it, a key that cannot leave
that TPM.
- The statement. A certified join returns
nodeKeyUri(AttestationResult.nodeKeyUri), and JoinService issues the certificate with two URI SANs: the SPIFFE ID first, thenurn:bc:tpm-ak:<nameAlg>:<digest>, the AK's TPM name in hex (tpmAkUri; the name covers the AK's whole public area, so only that key matches it). A join without a certified key (any other attestor, ornodeKey: 'any'without a certification) gets no statement and renews as before. Since 0e42018 JoinService writes a statement only when renewals can prove it again (JoinCrypto.knowsKeyStatement;pkiJoinCryptoknowsurn:bc:tpm-ak:alone): a join whose attestor returns any othernodeKeyUriis refused withnode-key-statement(security.join.refusedreasonjoin:node-key-statement) rather than issued a certificate that could never renew. - Where Renew reads it. From the presented leaf, which must already be byte for byte compute's own
node_certsrow, never from the CSR. A CSR that asks for no SAN at all still gets the statement back, and a CSR cannot remove it. - The proof. A leaf with a
urn:bc:tpm-ak:statement renews only withRenewRequest.key_attestation(field 5,TpmKeyAttestation): the AK's public area (ak_public), the new key's public area (key_public), and the AK'sTPM2_Certifyof that key (certify_info,signature) qualified bysha256(csr).verifyTpmKeyAttestationchecks that the AK's TPM name is the one in the statement and that it has attestation-key attributes, then runs the same checks as the join's node-key proof: the AK's signature, the qualifying data, the certified name,fixedTPM,fixedParent,sensitiveDataOriginand a signing key, and that the key's SPKI is the CSR's. - No downgrade. A missing or failing attestation refuses the renewal with
ACCESS_DENIED("Renew refused"), and JoinService logsrenew refusedwith the node id and the reason. Since 0e42018 it also publishessecurity.renew.refusedto the node owner, naming the node (see refused renewals). The new certificate carries the statement forward, so the next renewal needs the AK again.
| Reason | Meaning |
|---|---|
key-attestation-missing | The leaf names an AK and the request has no key_attestation |
key-attestation-ak | ak_public is not the named AK (another TPM's), or is not an attestation key |
node-key-malformed, node-key-signature, node-key-nonce, node-key-name, node-key-attributes, node-key-not-csr | The certification fails the same check as at join (a replayed certification for another CSR is node-key-nonce) |
tpm-ak-uri | The statement in the leaf is malformed |
key-statement-unknown | The leaf carries a urn:bc: statement this control plane does not know |
key-statement-unsupported | The JoinCrypto has no verifyKeyStatements (pkiJoinCrypto has one) |
The node certificate's URI SANs are now exactly one SPIFFE ID followed only by urn:bc: statements (at most four
URIs in all). The identity readers (nodeSpiffeIdOf in @better-compute/pki, used for mTLS identity, proofs of
possession, join responses and the gateway's renew relay) and bcd's pki.NodeIdentity take the node id from the
one SPIFFE ID and never from a statement; see PKI.
bcd's side. At each renewal renewOnce reads the current leaf. When it names an AK, bcd loads that AK alone
(tpm.Certifier, a primary of the endorsement hierarchy, no EK), checks that its TPM name is the one the certificate
names, derives the next node key in the same TPM connection, and has the AK certify it over sha256(csr) into
key_attestation. A root bcd node does this on the TPM it opened; an unprivileged one under [helper] asks
bcd helper (tpm.ak, tpm.nodekey, tpm.certify; 8870b8d), which holds the TPM connection, and checks that the
helper certified with the AK it announced (see the node key in the TPM). There is no fallback: where a certificate without a statement would get a file
key with a warning, this one fails the renewal instead and sends nothing.
| At renewal | What bcd does |
|---|---|
| The TPM cannot be opened | renewal: the certificate's node key is in the TPM, which is unavailable: ... |
| The TPM derives another AK | renewal: this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join (a1c1a46) |
| The TPM cannot make or certify the next key | renewal: the next node key in the TPM: ... or renewal: the TPM's attestation key certifying the next key: ... |
Each failure is logged as renewal failed with the attempt and expiresIn, and retried with backoff from
30 seconds to 10 minutes. Retries do not help a TPM that lost the AK or the key: the certificate (at most
24 hours) runs out, and the node goes offline.
Recovering a node whose TPM changed
A tpm-joined node recovers by joining again with its TPM. bcd join alone joins only when the old identity no
longer loads; one that still loads answers already joined as <node> whatever state its certificate is in. Since
a1c1a46 bcd join --reset --yes moves the old identity aside first, so the recovery command is the same whichever
case you are in:
# as root on the machine
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm--reset renames identity.json, node.key and a pending node.key.next to <name>.reset-<UTC time> in the data
directory (nothing is deleted) and the join then runs as on a new machine; see
joining again with --reset for its guards. The attestor keys the
machine on the SHA-256 of its EK, so what happens depends on whether the EK survived:
- The EK is the same (for example after
TPM2_Clear, which resets the owner hierarchy's seed but not the endorsement seed the EK and AK come from). The node key is a primary of the owner hierarchy, so the samenode.keyrecord now derives another key, and bcd no longer loads its identity: "the node key does not match the node certificate: the TPM derives another key from node.key (was the TPM cleared?); join again: bcd join --reset --yes, then bcd join" (since a1c1a46 this hint is on the key mismatch, where a clear shows; the AK survives a clear). The join finds no usable identity, and the attestor reclaims the node that EK is bound to: the same node id, a new identity generation (every older certificate revoked), a new certified key and a fresh AK statement. Here a plainbcd join --attestor tpmwould work too, since the identity no longer loads;--resetkeeps the old files instead of overwriting them. - The EK is new (another TPM, a replaced board, a reset endorsement hierarchy). The renewal's AK check fails
("another TPM, or its endorsement seed changed"). The attestor does not know the EK, but the old node still
exists: it holds the binding to the old EK and, usually, the machine's trusted hardware key, and a join whose
hardware key another node holds is refused (
reclaim), since atpmjoin carries noallowReclaim. Remove the old node first, then join again:- cordon the old node (
better-compute nodes cordon,nodes.cordon) and wait until it holds no open allocation; - delete it (
better-compute nodes delete <node> --yes, 4049621;nodes.delete, which since 54cad9e accepts a cordonedadoptednode): its certificates are revoked and its EK binding and trusted hardware key are freed (see removing an adopted host); - on the machine, as root, run
bcd join --reset --yes --attestor tpm. The machine joins wherever yourresolveorenrollmentsends it, as a new node (with neither, it is refused). It has a new node id; the--resetmoved the old node'sjournal.dbaside with the identity (b4fcc2e), sobcd nodestarts with a journal of its own.
- cordon the old node (
bcd join --reset without --yes moves nothing: it prints what it would move and exits 2. The hints bcd prints
spell the recovery as two runs, bcd join --reset --yes, then bcd join; one run with both does the same, because
the reset happens before the join reads any credential. If the join after a reset fails, the identity stays moved
aside and a later bcd join joins.
A node that joined with a token has no statement, so its renewal does not depend on the TPM's AK. Its node key, if
in the TPM, still needs that TPM to load: after a clear it joins again with a token, and a token join of a known
machine needs allowReclaim (see placeholder hardware and reclaims).
bcd with its TPM
tpmProducer(SoftwareTpm) is the agent side in TypeScript tests. bcd has its own, internal/tpm (f61b32d): a pure-Go
TPM 2.0 client over the kernel resource manager (/dev/tpmrm0, Linux only) that derives the default RSA-2048 EK and a
restricted P-256 AK and builds this attestor's { ek, ak } evidence and its round-2 answer (the
ActivateCredential secret and a quote with extraData = sha256(secret || sha256(csr))), with a software TPM,
tpmsim, for tests. Since 8545688 bcd join --attestor tpm uses it:
bcd join --config /etc/bcd/config.toml --attestor tpm- The TPM is the credential. No token is read or sent:
BCD_JOIN_TOKENis dropped from the environment, and--token-filewith--attestor tpmis a usage error. Where the machine joins is the attestor's decision (the bound node,resolveorenrollmentabove). - Which TPM. A host build opens
/dev/tpmrm0; a fakehw build keeps a software TPM in<data_dir>/fakehw-tpm.json, the same EK on every join of that machine. - The rounds.
agent.Joinsends the evidence, answers up to 3 challenge rounds for the same CSR, and waits out a pending approval (enrollment) before a fresh challenge round. A machine that already joined answersalready joined as <node>without touching the TPM, unless--reset --yesmoved its identity aside first (recovering a node whose TPM changed). - The node key, certified (cf24876). When node keys can use a TPM on this host (below),
agent.Joinmakes the join's node key in the attestor's own TPM (tpm.Attestor.HoldNodeKey, through theNodeKeyHolderinterface), and the round-2 answer carries the AK'sTPM2_Certifyof it oversha256(csr). If the TPM cannot make that key, the join stops with an error rather than falling back to a file. When node keys cannot use the TPM (a fakehw build withoutBCD_FAKEHW_TPM_KEYS=1), the key is a file, the answer carries no certification, and an attestor left atnodeKey: 'certified'refuses the join.bcd joinruns as root and opens the TPM itself,[helper]or not; since 8870b8d it no longer checks whether the helper's user can open/dev/tpmrm0, because that user never does (below). - The install script.
daemon install-script --attestor tpm(a006d9f) writes a script that reads no token, fails early without/dev/tpmrm0, and runsbcd join --config /etc/bcd/config.toml --attestor tpm < /dev/nullbefore (re)starting the units, so the same script installs and upgrades.
The node key in the TPM
Independent of the attestor, bcd keeps its node key in the host's TPM when there is one (965c14d, 50e18f9,
0ba2a79). The node key signs the mTLS handshake, CSRs, renewal proofs, node statements (attested reports, command
results, discovery assertions, secret and workload-token requests) and the WebSocket proof; in bcd it is a Go
crypto.Signer, whatever holds it, and the wire does not change: the same P1363 signatures and the same kid
(SHA-256 of the key's SPKI).
- The key.
tpm.NodeKeyTemplateis an unrestricted P-256 signing key (fixedTPM,fixedParent,sensitiveDataOrigin,userWithAuth,noDA,sign), a primary of the owner hierarchy derived from a 32-byte per-key unique. Nothing is stored in or evicted to the TPM: bcd keeps the unique and derives the same key again at every load. Each signature uses ECDSA with the digest's hash (SHA-256, -384 or -512, for TLS). - On disk.
node.key(andnode.key.nextduring a renewal) then holds only the record: a PEM block of typeBCD TPM NODE KEYwhose bytes are the unique, mode 0600. A key in the TPM does not load without that TPM. - When. Every new node key is made in the TPM when there is one: at
bcd joinand at every renewal's next key (under[helper], in the helper's TPM connection). A host build uses/dev/tpmrm0when it exists; a fakehw build uses its software TPM only withBCD_FAKEHW_TPM_KEYS=1. When the TPM cannot hold the key, bcd logs a warning and makes a PKCS#8 file key instead, except at the renewal of a certificate that names an AK, which fails rather than leave the TPM (see renewing a TPM-held key). A node that joined with a file key moves into the TPM at its next renewal. - Under
[helper](8870b8d, security review 3). The unprivilegedbcd nodenever opens the TPM. The kernel's resource manager filters no command, so whoever can open/dev/tpmrm0can change the hierarchies' authorization, clear the TPM through its (empty) lockout authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys) and attestation.bcd joinandbcd helperrun as root and open the TPM themselves; an unprivilegedbcd nodereaches its node keys only through the helper'stpm.*operations, which name a key by its 32-byte unique, and checks every signature the helper returns against the key's public half. The helper builds every TPM command itself, so the node picks no template, handle, hierarchy or authorization (see node keys through the helper). Install scripts from 0ba2a79 until 8870b8d put thebcduser in thetssgroup instead; an upgrade run takes it out again (see running bcd node unprivileged). - What it protects. Such a key cannot be copied off the machine, but anyone with the owner hierarchy's authorization on that TPM (root on the node) can derive and use it: non-exportable, not root-proof.
- Seeing it.
bcd doctorhas anode keycheck:ok"in the TPM",okfor a file key on a host with no TPM, andwarnfor a file key on a host with a TPM ("the next renewal moves it into the TPM (through bcd helper when bcd node runs unprivileged)"). Under[helper]itstpmcheck (e6c3f53) fails while the node's user is still in thetssgroup.bcd statusprintsnode key in the TPMornode key a fileon its identity line, and its JSON carriesnodeKey(tpmorfile).
What the control plane knows about it depends on how the key was made:
| When | Proven to the control plane | Where it shows |
|---|---|---|
A tpm join (cf24876) | Yes: the AK certifies the key (CertifyNodeKey, TPM2_Certify over sha256(csr)) and tpmAttestor checks it, refusing the join under the default nodeKey: 'certified' when it is missing | nodeKey: 'tpm' in node.joined and the node.joined or node.reclaimed audit row; urn:bc:tpm-ak: in the certificate (d94f1f3) |
| A join with any other attestor | No: bcd still makes the key in the TPM when it can, but sends no certification | nodeKey: 'unattested'; no statement in the certificate |
| A renewal of a certificate that names an AK (d94f1f3) | Yes: the same AK certifies the next key (RenewRequest.key_attestation), and JoinService refuses the renewal without it | the statement, carried into the new certificate and its node_certs row |
| A renewal of any other certificate | No: the next key is made in the TPM when bcd can open one, but nothing proves it | nothing; the certificate and node_certs do not record where the key lives |
So for a certified tpm join the proof lasts: every certificate the node holds from then on names a key that its
TPM made and certified. For every other node the control plane has only bcd's word (bcd status and bcd doctor
report it locally).
What is tested, and with what TPM:
| Test | TPM | What it proves |
|---|---|---|
TestTSContract (daemon/internal/tpm) | tpmsim fixture, with a manufacturer EK certificate in NV (b973e05) | bcd's round-1 evidence is still byte for byte the fixture's, including the EK certificate bcd reads from NV index 0x01C00002 (tpm.EKCertIndex) and sends as ekCert, and the challenge tpmAttestor made for it activates and quotes through bcd's client (TypeScript to Go); since cf24876 the answer also carries the AK's certification of a node key held in the sim (answer.nodeKey) |
tests/pki-attestors-tpm.test.ts | tpmProducer over a SoftwareTpm whose nodeKey it certifies as bcd does | under nodeKey: 'certified', a certified CSR key is admitted with nodeKey: 'tpm'; no certification is node-key-not-certified, and a certified key other than the CSR's is node-key-not-csr; nodeKey: 'any' admits the uncertified answer without a nodeKey claim |
tests/pki-attestors-tpm-bcd.test.ts | the same fixture | tpmAttestor with ekRoots: [ek-root.der] and no allowUncertifiedEk verifies the EK certificate's chain, challenges bcd's evidence and admits bcd's answer, with its node key certified (nodeKey: 'tpm'), while a CSR key other than the certified one is node-key-not-csr; it refuses the EK under another root (ek-certificate-chain), and the answer for another CSR (challenge-for-another-csr) or after the challenge expired (challenge-expired) (Go to TypeScript) |
tests/e2e/bcd-tpm.test.ts (BCD_E2E=1, WSL or Linux; db9bd46, 8a9af54) | tpmsim in a fakehw bcd, with an EK certificate from a manufacturer root the test makes (BCD_FAKEHW_TPM_EK_CA) and the node key in that TPM (BCD_FAKEHW_TPM_KEYS=1) | a real bcd join --attestor tpm with no token against the dev stack's JoinService, whose tpmAttestor trusts only that root (ekRoots, no allowUncertifiedEk): two rounds, resolve seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in the resolved pool and bound to sha256(EK) in node_bindings; node.key a BCD TPM NODE KEY record; the node key certified by the AK, so node.joined says nodeKey: 'tpm' for this join and for the rejoin below (cf24876); with its identity removed (a reinstall) the same TPM joins as the same node; bcd node on that identity, signing its session and statements in the TPM, is adopted; since d94f1f3 the certificate carries the AK's urn:bc:tpm-ak: statement, and bcd node renews (BCD_RENEW_AFTER=3s) with the AK certifying the next key, which stays a BCD TPM NODE KEY record while the new certificate keeps the statement |
tests/server-join-tpm-renew.test.ts (d94f1f3) | tpmProducer and softwareTpmCertify over SoftwareTpm | a certified join names the AK; a renewal without key_attestation is key-attestation-missing, another TPM's AK key-attestation-ak, a certification of another key node-key-not-csr, one for another CSR node-key-nonce; a good renewal keeps the statement although its CSR asks for no SAN, and the next renewal needs the AK again; an uncertified join (nodeKey: 'any') gets no statement and renews without one |
internal/agent TestRenewCertifiedByTheAK (d94f1f3) | tpmsim | a certificate naming the sim's AK renews with the next key made in that TPM and certified by it; with no TPM the renewal fails (unavailable), and with another TPM (another AK) it fails before any request, never falling back to a file key |
internal/helper TestHelperTPMKeys (8870b8d, Linux) | tpmsim behind a helper server on a test socket | the key and AK the node gets through tpm.* are the same as the TPM's own; signing and certifying; at most 8 keys loaded (least recently used unloaded) and a released key derived again on use; refusals of bad shapes, unknown verbs and a helper without a TPM |
scripts/daemon-tpm-fixture.mjs regenerates the fixtures in daemon/internal/tpm/testdata after a deliberate change
of either side, so a format change fails a test rather than a machine's join. Since b973e05 the contract runs the
attestor as against a real TPM: ek-root.der is a test manufacturer root and ek-cert.der the EK certificate it
issued for the sim's EK (made once by the Go test, and kept with sim.json when the fixtures are regenerated), and
the attestor verifies that chain through ekRoots. Since 8a9af54 the dev-stack e2e does the same: a fakehw bcd
given BCD_FAKEHW_TPM_EK_CA (a PEM file with the root's certificate and its PKCS#8 key) issues its software TPM an EK
certificate in NV at each load, so allowUncertifiedEk is no longer used by any bcd test. Every test uses a software
TPM: no test has run against a hardware TPM, a real manufacturer's EK certificate or a real /dev/tpmrm0.
Single-use evidence
Tokens are not the only thing that must not be used twice. cloud-iid and discovery record what they
accepted in attestation_uses (attestor, subject, expires_at) inside the join transaction:
| Attestor | Subject | Kept until |
|---|---|---|
cloud-iid | SHA-256 of provider, account and instance | forever |
discovery | SHA-256 of the signed assertion body | the assertion's expiry |
A second join with the same subject is refused. pruneAttestationUses(tx, { batch? }) from
@better-compute/postgres deletes expired rows, 5,000 per call by default; see
retention. Single-use tokens (provisioned, discovery) must have exactly one
use left, so a token minted for several machines is never accepted as a machine credential.
Two-round joins
Some attestors cannot decide in one request: tpm must challenge the TPM, and discovery or tpm may wait for
an operator. Such an attestor returns an AttestationDeferral instead of a result:
interface AttestationDeferral {
deferred: 'approval' | 'challenge';
requestId?: string; // approval: the join request
retryAfterSeconds?: number; // approval: default 30
challenge?: Uint8Array; // challenge: attestor-specific bytes
notAfter?: Date; // challenge: answers after this are refused
}JoinService then commits the transaction (so the attestor's own rows, such as a pending join request, persist),
issues nothing and spends nothing, and answers with a signed JoinResponseBody whose deferral holds either
JoinPending{ request_id, retry_after_seconds } or JoinChallenge{ attestor, payload, not_after }. The node id
and chain are empty, and csr_sha256 still binds the answer to the request. The agent re-sends Join with the
same CSR: after the wait, or with its answer to the challenge in attestation. Attestors therefore consume
tokens and record uses only on the final round.
On the agent side, openJoinResponse reads a response as joined, pending or challenge, and verifyJoinResponse
refuses a deferral. runAgentJoin in @better-compute/testing is the reference retry loop.
Requiring an attestor
By default a pool takes a machine whichever attestor admitted it. Since 1516728 a pool can require hardware
attestation with policy.joinAttestors, the ids of the attestors that may admit a machine into it:
const pool = await client.pools.get({ tenantId: 'root', poolId: 'pool_01k…' });
await client.pools.update({
tenantId: 'root',
poolId: pool.id,
// The policy is replaced as a whole, so keep the rest of it.
policy: { ...pool.policy, joinAttestors: ['tpm'] }, // or ['tpm', 'x509pop'], or ['join-token', 'tpm'] to keep tokens
});- Where it is checked. JoinService checks it in the transaction that resolves the pool, before any reclaim or
insert. A join or a reclaim admitted by an attestor that is not listed is refused with
ACCESS_DENIED, reasonpool-attestor. A token join isjoin-token, so a pool that lists onlytpmrefuses tokens, and a token reclaim of a TPM-bound machine (allowReclaimwith the same trusted hardware key) is refused while its TPM still reclaims it. - Who sees the refusal. The attestor has already verified the machine, so the refusal is attributed to the pool's
owner:
security.join.refusedwith the attestor and reasonjoin:pool-attestor, and the token or node id when known. - Validation.
pools.createandpools.updaterefuse (INVALID_INPUT) a value that is not an array of 1 to 16 ids of attestors this control plane runs:join-tokenand every attestor registered inattestorsor by a module or plugin. - Moving nodes. No API moves a node between pools. A node gets its pool at join, where the check runs, or at
metal.enroll, whose machine gets its identity only through JoinService.
Tested by tests/server-join-pool-attestors.test.ts.
Choosing an attestor
| Situation | Use |
|---|---|
| A server you install by hand or with configuration management | join-token, with expect when the firmware reports placeholder identifiers |
| A node whose certificate expired while it was offline | x509pop |
| The OS a Better Compute deploy or a provider bootstrap installed | provisioned |
| A ramdisk during bare-metal commissioning | discovery, with approval if unexpected machines should wait for a person |
| A VM your provider account created | cloud-iid |
| Hardware identity that must survive reinstalls, on TPM 2.0 machines | tpm (v0.5), with bcd join --attestor tpm; add policy.joinAttestors: ['tpm'] to refuse every other way in |
To write your own attestor, see attestors.
Better Compute is created by Sean Filimon
Last updated
Joining
Join tokens, the join flow, node certificates and renewal, adopting a brownfield server with the install script, and the join request approval queue.
Inventory and health
NodeInventory v1 and its section hashes, snapshots and diffs, GPU naming, devices and atoms, conditions, XID handling, remediation and acceptance.