# Attestation (/docs/guides/nodes/attestation) > The attestors that admit machines (join-token, x509pop, provisioned, discovery, cloud-iid, tpm), what each proves, single-use evidence and two-round joins. A join token proves that someone with access to your API wanted a machine to join. That is enough for a server you install by hand, but not for a machine that PXE-boots on its own, a cloud VM, or a server whose certificate expired over a long weekend. An attestor answers the question for one kind of evidence: is this machine allowed to join, and as which node? Each attestor is a `@better-compute/core` `AttestorVerifier` that runs inside JoinService's join transaction. `JoinRequest.attestor` names it (empty means `join-token`), `JoinRequest.attestation` carries its evidence, and a sealed token, when the request has one, is opened and handed over as `evidence.token`. Every refusal reaches the agent as the same `ACCESS_DENIED`; the reason goes only to the `onRefusal` hook and the `security.join.refused` event (see [joining](/docs/guides/nodes/joining)). ## The attestors [#the-attestors] | Attestor | Phase | Roles | Evidence | Proves | Joins as | | ------------- | ----- | --------------------- | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------- | | `join-token` | v0.1 | `node`, `site` | a sealed `node` or `site` token | the holder of a live token for this pool or site | a new node, or a reclaim with `allowReclaim` | | `x509pop` | v0.2 | `node`, `site` | its last certificate chain and a proof by that key | the machine still holds the key of a recent certificate | the same node, in place | | `provisioned` | v0.2 | `node` | a sealed single-use `provisioned` token and a product UUID claim | the OS a deploy or provider bootstrap installed | the target node re-keyed, or a new node | | `discovery` | v0.2 | `ramdisk` | a sealed single-use `discovery` token and a signed discovery assertion | the ramdisk the site's boot server just booted | a ramdisk identity of at most 2 hours | | `cloud-iid` | v0.3 | `node` (configurable) | a provider-signed instance identity | the cloud instance your provider account created | whatever `resolve` answers | | `tpm` | v0.5 | `node` (configurable) | TPM endorsement and attestation keys, then an activated credential, a quote and the AK's certification of the node key | the TPM holding that endorsement key, and (by default) that the same TPM holds the CSR's key | the node bound to that key | Only `join-token` is built into the server. You add the others, and one with id `join-token` replaces the built-in: ```ts title="compute.ts" import { betterCompute, pkiJoinCrypto } from 'better-compute/server'; import { discoveryAttestor, provisionedAttestor, x509popAttestor, } from 'better-compute/pki'; import { discoveryWorkflowNodeResolver, provisionedTargetResolver, } from 'better-compute/provision'; // `bundle`: the PKI bundle `better-compute pki init` wrote, loaded with importPkiBundle(text, { secret }). export const compute = betterCompute({ // database, secret, baseURL, iam, modules, … pki: { nodeCa: bundle.nodeCa, join: pkiJoinCrypto({ trustDomain: bundle.trustDomain, nodeCa: bundle.nodeCa, signingKey: bundle.joinSigning, encryptionKey: bundle.joinEncryption.privateKey, }), }, attestors: [ x509popAttestor(), provisionedAttestor({ resolveTarget: provisionedTargetResolver() }), discoveryAttestor({ resolveWorkflowNode: discoveryWorkflowNodeResolver(), approval: 'unknown-machines', }), ], }); ``` The attestors from `@better-compute/pki` also take `onRefusal(refusal)`, called with `{ attestor, reason, tokenId?, nodeId?, tenantId? }` for your own audit or metrics. ## join-token [#join-token] The v0.1 attestor, described in [joining](/docs/guides/nodes/joining). It locks the `join_tokens` row, checks the secret in constant time, revocation, expiry at `compute_now()`, the remaining uses and the role (`node` tokens admit `node`, `site` tokens admit `site`), and consumes one use. JoinService enforces the token's `expect` binding and reclaim flag for every token-based attestor, not only this one. ## x509pop [#x509pop] A node's certificate lasts 24 hours. A machine that was powered off or cut off longer than that comes back with an expired certificate, and renewal refuses expired certificates. `x509pop` lets it re-join as itself: it presents its last certificate chain and a proof by the certificate's key, bound to its new CSR. The evidence is UTF-8 JSON: ```json title="JoinRequest.attestation" { "v": 1, "nodeId": "node_…", "chain": ["", ""], "clientTime": 1790000000000, "sig": "" } ``` `sig` is an ES256 signature under purpose `x509pop` over the CSR's SHA-256, the node id and the client time, so neither a Renew proof nor an earlier `x509pop` proof can be replayed as one. The attestor refuses unless: * the leaf is byte for byte a `node_certs` row of that node, of its current identity generation, and not revoked; * the node is not `retired`, and the certificate's role is the requested one (`node` or `site`; a lessee's `node-tenant` certificate re-joins as `node` and JoinService restores the restriction); * the certificate expired at most `graceMs` ago (7 days by default); * `clientTime` is within `windowMs` (60 seconds) of database time, and the proof was not used before (`replayCache`, in memory by default, 10,000 entries). The result names the node, so JoinService reclaims it in place: it raises the identity generation, revokes every older certificate and keeps the node's state. A revoked certificate never re-joins; neither does one from before `nodes.revokeIdentity`. Agents build the evidence with `createX509popEvidence` (or `x509popProducer` for `FakeDaemon`). bcd does not produce this evidence yet: it joins with `join-token` only and answers no attestor challenge, so a bcd node whose certificate expired offline cannot re-join this way yet. ## provisioned [#provisioned] When Better Compute deploys an operating system, the deploy renders a single-use `provisioned` token into the machine's user data (the provisioning engine's `userdata.render` step). The installed system joins with it. The attestor requires: * a token of role `provisioned` with exactly one use left (`not-single-use` otherwise); * a product UUID in the machine's claims (`no-product-uuid`); * a binding to one machine: the token's `expect.productUuid`, or a target the host resolves (`unbound`). `resolveTarget(tx, token)` answers with the node the token was minted for and its expected product UUID. `provisionedTargetResolver()` from `@better-compute/provision` finds the running workflow whose `userdata.render` step minted the token, and reads the product UUID from that node's newest inventory. With a target node, the join re-keys that node in place and keeps its provisioning state; a UUID that differs is `product-uuid-mismatch`. Without one, the machine joins as a new node, which is what a provider bootstrap such as Hetzner Robot's installimage does. ## discovery [#discovery] During commissioning, a bare-metal server PXE-boots a platform ramdisk before anything is installed on it. The ramdisk has no identity of its own, so two parties vouch for it: the iPXE script carries a single-use `discovery` token bound to the MAC, and the site's boot server signs a statement that it just served that MAC. The evidence is the sealed token plus a serialized `SignedDiscoveryAssertion` (`bettercompute.site.v1`). The assertion names the site, the vouching site agent's node id, the boot lease epoch it held, the MAC, and optionally the IP, product UUID and workflow id, with `issued_at` and `not_after`. The attestor checks: * the token: role `discovery`, one use left, issued for the assertion's site, and its `expect.mac` (when set) equal to the asserted MAC; * the signer: a `site` node of that site, not retired, currently holding the `site::boot` lease at exactly the asserted epoch, so a fenced boot server can no longer vouch; the signature (purpose `discovery-assertion`) must verify against its unexpired certificate; * freshness: the assertion is valid until the earlier of its `not_after` and 15 minutes after `issued_at` (`maxAgeMs`); * the machine: its claimed MACs include the asserted MAC, and its product UUID matches when the assertion names one. Each signed assertion admits one join (`assertion-reused` otherwise), and the token is spent. The resulting certificate is a `ramdisk` certificate of at most 2 hours (`certTtlSeconds` may shorten it) that allows only the commissioning command kinds, `discoveryAllowedKinds` by default: `inventory.refresh`, `logs.collect`, `diag.*`, `disk.wipe`, `deploy.*` and `boot.kexec`, which are exactly the registered kinds whose roles include `ramdisk` (`tests/pki-attestors.test.ts` checks the two agree). Pass `allowedKinds` to change the list. | Option | Default | Effect | | ------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `resolveWorkflowNode(tx, workflowId)` | none | A known machine re-commissions as its node (`discoveryWorkflowNodeResolver()` answers for running workflows). Without it every discovery creates a node. | | `approval` | `'never'` | `'unknown-machines'` defers machines no workflow names; `'always'` defers every one. | | `hardwareKey(claims)` | none | The trusted hardware key requests are keyed on; without one they key on site and MAC. | | `pendingTtlSeconds` | 86,400 | How long a pending request waits for a decision. | | `retryAfterSeconds` | 30 | When the ramdisk retries a deferred join. | ### Both halves in bcd [#both-halves-in-bcd] bcd produces this evidence on both sides. `bcd site` signs one assertion for every iPXE script its boot server renders, while it holds the boot lease: the site id, its own node id, the lease epoch, the MAC, the IP (from iPXE's query, else the connection), the product UUID (from iPXE, else the boot entry), the workflow, `issued_at` in server time, and a `not_after` of at most 15 minutes and never past the boot entry's, signed by its node key under purpose `discovery-assertion`. It fetches the machine's discovery token with a `SecretRequest` bound to the workflow and puts both on the ramdisk's kernel command line (`bc.token`, `bc.assertion`). `bcd ramdisk` reads them back, seals the token to the pinned JoinService key and sends `Join` with attestor `discovery`, role `ramdisk`, the serialized `SignedDiscoveryAssertion` as the attestation, and its hardware claims (MACs, product UUID). A deferral is retried with the same CSR. The Go tests check the assertion's fields and signature, and the TypeScript vectors pin the signing input, but no test yet joins a real `bcd ramdisk` through this attestor. See [the boot server](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site) and [the ramdisk](/docs/daemon/install#the-ramdisk). ## cloud-iid [#cloud-iid] A cloud VM that your provider account created can prove it with the identity document its cloud signs, so it needs no token at all. `cloudIidAttestor({ providers, resolve, roles? })` verifies the document with one of its providers, then asks `resolve(tx, identity, { role, claims })` where the instance joins. `resolve` returns an attestation result (tenant, pool, site, and `nodeId` for a known machine) or `undefined` to refuse (`unknown-instance`), for example for an instance in an account you do not manage. ```json title="JoinRequest.attestation" { "provider": "gcp", "document": "" } ``` | Provider | Verifies | Bound to the CSR | | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `gcpInstanceIdentity({ audience, fetch or jwks, projects?, maxAgeMs? })` | A Compute Engine identity token with `format=full`, signed by Google (RS256 or ES256, JWKS cached an hour). `projects` limits the accepted project ids. | Yes: the agent requests audience `cloudIidAudience(audience, sha256(csr))`, that is `#`. | | `awsInstanceIdentity({ certificates, maxAgeMs?, accounts? })` | The EC2 identity document and its `signature`, checked with the region's AWS certificate, which you supply (`'*'` covers regions not listed). `accounts` limits the account ids. | No: accepted only within `maxAgeMs` (30 minutes) of the instance's `pendingTime`. | | `oidcInstanceIdentity({ id, issuers, jwks, audience, identity })` | Any issuer that signs JWTs into a JWKS; `identity(claims)` maps the claims to account and instance. | Yes, through the audience. | Identity tokens older than `maxAgeMs` (10 minutes by default for OIDC and GCP) are refused. Whatever the provider, each (provider, account, instance) admits exactly one join, ever (`instance-reused`): an AWS document is readable by any process on the VM, and even a bound token must not mint a second node. A known instance that lost its certificate re-joins with `x509pop`. ## tpm [#tpm] `tpm` (v0.5) ties a node to its TPM's endorsement key (EK), an identity that survives a reinstall or a disk swap. It follows the go-attestation enrollment flow as a two-round join. 1. **Challenge.** The agent sends its EK (public area and manufacturer certificate) and an attestation key (AK). The attestor checks the EK certificate chains to one of `ekRoots` and that the AK has attestation-key attributes, then answers with a `MakeCredential` challenge plus an opaque state sealed under `challengeKey` (AES-256-GCM, bound to the CSR, at most 5 minutes). It writes no row, so any `api` replica can finish the join. 2. **Answer.** Only the TPM holding that EK, with that AK loaded, can activate the credential. The agent returns the secret and a quote of its PCRs whose `extraData` is `sha256(secret ‖ sha256(csr))`. The attestor checks the secret, the quote signature, the nonce, that the PCR values match the quote's digest, and your `pcrPolicy`. 3. **Node key** (cf24876). The same answer carries `nodeKey { public, certifyInfo, signature }`: a `TPM2_Certify` of the node key by the AK, qualified by `sha256(csr)`. The attestor checks that the certification is signed by the AK it just proved beside the EK, that it is a `TPM_ST_ATTEST_CERTIFY` whose `extraData` is `sha256(csr)`, that the certified name is the TPM name of `public`, that the key was made in the TPM and can never leave it (`fixedTPM`, `fixedParent`, `sensitiveDataOrigin` and `sign` set, `decrypt` clear), and that its SubjectPublicKeyInfo is the CSR's (JoinService passes its SHA-256 as `csrSpkiSha256`). A certification that fails a check is refused with `node-key-malformed`, `node-key-signature`, `node-key-nonce`, `node-key-name`, `node-key-attributes` or `node-key-not-csr`. With the default `nodeKey: 'certified'`, an answer without one is refused too (`node-key-not-certified`); `nodeKey: 'any'` admits it, and the result then makes no claim about the key. It then decides where the machine joins, keyed by the SHA-256 of the EK: the node that key is already bound to (a reclaim), else your `resolve` allow-list (for example a vendor's delivery manifest), else operator `enrollment` through [join requests](/docs/guides/nodes/joining). JoinService binds the EK to the node (`node_bindings`): a later join proving it reclaims that node, and it can never admit another one (`bound-elsewhere`). When the node key was certified, the result carries `nodeKey: 'tpm'`, and JoinService records it in the `node.joined` event and in the join's audit row (`nodeKey`: `tpm`, or `unattested` for every other join; see [joining](/docs/guides/nodes/joining#the-join-flow)). Since d94f1f3 the result also carries `nodeKeyUri`, the AK's name as `urn:bc:tpm-ak::`, which JoinService writes into the certificate so that every renewal must be certified by the same AK (see [renewing a TPM-held key](#renewing-a-tpm-held-key)). | Option | Meaning | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `challengeKey` | 32 bytes shared by every `api` replica; seals the challenge state. Required. | | `ekRoots` | Manufacturer EK CA certificates (PEM or DER). | | `allowUncertifiedEk` | Accept an EK without a certificate (swtpm, labs). Never in production. | | `pcrSelection` | PCRs the quote must cover; default the SHA-256 bank, PCRs 0 to 7. | | `pcrPolicy(pcrs)` | Returns a refusal reason, or `undefined` to accept. | | `resolve(tx, identity, { role })` | Where an unbound EK joins, or `undefined`. | | `enrollment` | `{ tenantId, siteId, poolId?, pendingTtlSeconds?, retryAfterSeconds? }`: defer unknown EKs for approval. | | `challengeTtlMs` | Challenge lifetime, at most 5 minutes. | | `nodeKey` | `'certified'` (the default): admit a join only when the AK certifies the CSR's key as one this TPM made and never lets out. `'any'`: also admit a node key held elsewhere, such as a file. | > **nodeKey defaults to certified.** Since cf24876 `nodeKey: 'certified'` is the default, so a `tpm` join whose agent keeps its node key outside the TPM is refused (`node-key-not-certified`), including a machine that joined with a file key before and joins again. Set `nodeKey: 'any'` to keep admitting those; their certificates then name no AK and renew without a TPM proof. ### Renewing a TPM-held key [#renewing-a-tpm-held-key] The join proves the first node key. Since d94f1f3 every renewal proves the next one the same way, so a certificate issued after a certified `tpm` join names, for its whole life and every renewal after it, a key that cannot leave that TPM. 1. **The statement.** A certified join returns `nodeKeyUri` (`AttestationResult.nodeKeyUri`), and JoinService issues the certificate with two URI SANs: the SPIFFE ID first, then `urn:bc:tpm-ak::`, the AK's TPM name in hex (`tpmAkUri`; the name covers the AK's whole public area, so only that key matches it). A join without a certified key (any other attestor, or `nodeKey: 'any'` without a certification) gets no statement and renews as before. Since 0e42018 JoinService writes a statement only when renewals can prove it again (`JoinCrypto.knowsKeyStatement`; `pkiJoinCrypto` knows `urn:bc:tpm-ak:` alone): a join whose attestor returns any other `nodeKeyUri` is refused with `node-key-statement` (`security.join.refused` reason `join:node-key-statement`) rather than issued a certificate that could never renew. 2. **Where Renew reads it.** From the presented leaf, which must already be byte for byte compute's own `node_certs` row, never from the CSR. A CSR that asks for no SAN at all still gets the statement back, and a CSR cannot remove it. 3. **The proof.** A leaf with a `urn:bc:tpm-ak:` statement renews only with `RenewRequest.key_attestation` (field 5, `TpmKeyAttestation`): the AK's public area (`ak_public`), the new key's public area (`key_public`), and the AK's `TPM2_Certify` of that key (`certify_info`, `signature`) qualified by `sha256(csr)`. `verifyTpmKeyAttestation` checks that the AK's TPM name is the one in the statement and that it has attestation-key attributes, then runs the same checks as the join's node-key proof: the AK's signature, the qualifying data, the certified name, `fixedTPM`, `fixedParent`, `sensitiveDataOrigin` and a signing key, and that the key's SPKI is the CSR's. 4. **No downgrade.** A missing or failing attestation refuses the renewal with `ACCESS_DENIED` ("Renew refused"), and JoinService logs `renew refused` with the node id and the reason. Since 0e42018 it also publishes `security.renew.refused` to the node owner, naming the node (see [refused renewals](/docs/operations/security#refused-renewals)). The new certificate carries the statement forward, so the next renewal needs the AK again. | Reason | Meaning | | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | `key-attestation-missing` | The leaf names an AK and the request has no `key_attestation` | | `key-attestation-ak` | `ak_public` is not the named AK (another TPM's), or is not an attestation key | | `node-key-malformed`, `node-key-signature`, `node-key-nonce`, `node-key-name`, `node-key-attributes`, `node-key-not-csr` | The certification fails the same check as at join (a replayed certification for another CSR is `node-key-nonce`) | | `tpm-ak-uri` | The statement in the leaf is malformed | | `key-statement-unknown` | The leaf carries a `urn:bc:` statement this control plane does not know | | `key-statement-unsupported` | The `JoinCrypto` has no `verifyKeyStatements` (`pkiJoinCrypto` has one) | The node certificate's URI SANs are now exactly one SPIFFE ID followed only by `urn:bc:` statements (at most four URIs in all). The identity readers (`nodeSpiffeIdOf` in `@better-compute/pki`, used for mTLS identity, proofs of possession, join responses and the gateway's renew relay) and bcd's `pki.NodeIdentity` take the node id from the one SPIFFE ID and never from a statement; see [PKI](/docs/operations/pki#trust-domain-and-spiffe-ids). **bcd's side.** At each renewal `renewOnce` reads the current leaf. When it names an AK, bcd loads that AK alone (`tpm.Certifier`, a primary of the endorsement hierarchy, no EK), checks that its TPM name is the one the certificate names, derives the next node key in the same TPM connection, and has the AK certify it over `sha256(csr)` into `key_attestation`. A root `bcd node` does this on the TPM it opened; an unprivileged one under `[helper]` asks `bcd helper` (`tpm.ak`, `tpm.nodekey`, `tpm.certify`; 8870b8d), which holds the TPM connection, and checks that the helper certified with the AK it announced (see [the node key in the TPM](#the-node-key-in-the-tpm)). There is no fallback: where a certificate without a statement would get a file key with a warning, this one fails the renewal instead and sends nothing. | At renewal | What bcd does | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The TPM cannot be opened | `renewal: the certificate's node key is in the TPM, which is unavailable: ...` | | The TPM derives another AK | `renewal: this TPM's attestation key is not the one the certificate names (another TPM, or its endorsement seed changed): the certificate cannot be renewed here; join again: bcd join --reset --yes, then bcd join` (a1c1a46) | | The TPM cannot make or certify the next key | `renewal: the next node key in the TPM: ...` or `renewal: the TPM's attestation key certifying the next key: ...` | Each failure is logged as `renewal failed` with the attempt and `expiresIn`, and retried with backoff from 30 seconds to 10 minutes. Retries do not help a TPM that lost the AK or the key: the certificate (at most 24 hours) runs out, and the node goes offline. #### Recovering a node whose TPM changed [#recovering-a-node-whose-tpm-changed] A `tpm`-joined node recovers by joining again with its TPM. `bcd join` alone joins only when the old identity no longer loads; one that still loads answers `already joined as ` whatever state its certificate is in. Since a1c1a46 `bcd join --reset --yes` moves the old identity aside first, so the recovery command is the same whichever case you are in: ```bash # as root on the machine bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm ``` `--reset` renames `identity.json`, `node.key` and a pending `node.key.next` to `.reset-` in the data directory (nothing is deleted) and the join then runs as on a new machine; see [joining again with --reset](/docs/daemon/install#joining-again-with---reset) for its guards. The attestor keys the machine on the SHA-256 of its EK, so what happens depends on whether the EK survived: * **The EK is the same** (for example after `TPM2_Clear`, which resets the owner hierarchy's seed but not the endorsement seed the EK and AK come from). The node key is a primary of the owner hierarchy, so the same `node.key` record now derives another key, and bcd no longer loads its identity: "the node key does not match the node certificate: the TPM derives another key from node.key (was the TPM cleared?); join again: bcd join --reset \--yes, then bcd join" (since a1c1a46 this hint is on the key mismatch, where a clear shows; the AK survives a clear). The join finds no usable identity, and the attestor reclaims the node that EK is bound to: the same node id, a new identity generation (every older certificate revoked), a new certified key and a fresh AK statement. Here a plain `bcd join --attestor tpm` would work too, since the identity no longer loads; `--reset` keeps the old files instead of overwriting them. * **The EK is new** (another TPM, a replaced board, a reset endorsement hierarchy). The renewal's AK check fails ("another TPM, or its endorsement seed changed"). The attestor does not know the EK, but the old node still exists: it holds the binding to the old EK and, usually, the machine's trusted hardware key, and a join whose hardware key another node holds is refused (`reclaim`), since a `tpm` join carries no `allowReclaim`. Remove the old node first, then join again: 1. cordon the old node (`better-compute nodes cordon`, `nodes.cordon`) and wait until it holds no open allocation; 2. delete it (`better-compute nodes delete --yes`, 4049621; `nodes.delete`, which since 54cad9e accepts a cordoned `adopted` node): its certificates are revoked and its EK binding and trusted hardware key are freed (see [removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host)); 3. on the machine, as root, run `bcd join --reset --yes --attestor tpm`. The machine joins wherever your `resolve` or `enrollment` sends it, as a new node (with neither, it is refused). It has a new node id; the `--reset` moved the old node's `journal.db` aside with the identity (b4fcc2e), so `bcd node` starts with a journal of its own. `bcd join --reset` without `--yes` moves nothing: it prints what it would move and exits 2. The hints bcd prints spell the recovery as two runs, `bcd join --reset --yes`, then `bcd join`; one run with both does the same, because the reset happens before the join reads any credential. If the join after a reset fails, the identity stays moved aside and a later `bcd join` joins. A node that joined with a token has no statement, so its renewal does not depend on the TPM's AK. Its node key, if in the TPM, still needs that TPM to load: after a clear it joins again with a token, and a token join of a known machine needs `allowReclaim` (see [placeholder hardware and reclaims](/docs/guides/nodes/joining#placeholder-hardware-and-reclaims)). ### bcd with its TPM [#bcd-with-its-tpm] `tpmProducer(SoftwareTpm)` is the agent side in TypeScript tests. bcd has its own, `internal/tpm` (f61b32d): a pure-Go TPM 2.0 client over the kernel resource manager (`/dev/tpmrm0`, Linux only) that derives the default RSA-2048 EK and a restricted P-256 AK and builds this attestor's `{ ek, ak }` evidence and its round-2 answer (the `ActivateCredential` secret and a quote with `extraData = sha256(secret || sha256(csr))`), with a software TPM, `tpmsim`, for tests. Since 8545688 `bcd join --attestor tpm` uses it: ```bash bcd join --config /etc/bcd/config.toml --attestor tpm ``` * **The TPM is the credential.** No token is read or sent: `BCD_JOIN_TOKEN` is dropped from the environment, and `--token-file` with `--attestor tpm` is a usage error. Where the machine joins is the attestor's decision (the bound node, `resolve` or `enrollment` above). * **Which TPM.** A host build opens `/dev/tpmrm0`; a fakehw build keeps a software TPM in `/fakehw-tpm.json`, the same EK on every join of that machine. * **The rounds.** `agent.Join` sends the evidence, answers up to 3 challenge rounds for the same CSR, and waits out a pending approval (`enrollment`) before a fresh challenge round. A machine that already joined answers `already joined as ` without touching the TPM, unless `--reset --yes` moved its identity aside first ([recovering a node whose TPM changed](#recovering-a-node-whose-tpm-changed)). * **The node key, certified** (cf24876). When node keys can use a TPM on this host (below), `agent.Join` makes the join's node key in the attestor's own TPM (`tpm.Attestor.HoldNodeKey`, through the `NodeKeyHolder` interface), and the round-2 answer carries the AK's `TPM2_Certify` of it over `sha256(csr)`. If the TPM cannot make that key, the join stops with an error rather than falling back to a file. When node keys cannot use the TPM (a fakehw build without `BCD_FAKEHW_TPM_KEYS=1`), the key is a file, the answer carries no certification, and an attestor left at `nodeKey: 'certified'` refuses the join. `bcd join` runs as root and opens the TPM itself, `[helper]` or not; since 8870b8d it no longer checks whether the helper's user can open `/dev/tpmrm0`, because that user never does (below). * **The install script.** [`daemon install-script --attestor tpm`](/docs/daemon/install#generate-the-script) (a006d9f) writes a script that reads no token, fails early without `/dev/tpmrm0`, and runs `bcd join --config /etc/bcd/config.toml --attestor tpm < /dev/null` before (re)starting the units, so the same script installs and upgrades. ### The node key in the TPM [#the-node-key-in-the-tpm] Independent of the attestor, bcd keeps its node key in the host's TPM when there is one (965c14d, 50e18f9, 0ba2a79). The node key signs the mTLS handshake, CSRs, renewal proofs, node statements (attested reports, command results, discovery assertions, secret and workload-token requests) and the WebSocket proof; in bcd it is a Go `crypto.Signer`, whatever holds it, and the wire does not change: the same P1363 signatures and the same `kid` (SHA-256 of the key's SPKI). * **The key.** `tpm.NodeKeyTemplate` is an unrestricted P-256 signing key (`fixedTPM`, `fixedParent`, `sensitiveDataOrigin`, `userWithAuth`, `noDA`, `sign`), a primary of the owner hierarchy derived from a 32-byte per-key unique. Nothing is stored in or evicted to the TPM: bcd keeps the unique and derives the same key again at every load. Each signature uses ECDSA with the digest's hash (SHA-256, -384 or -512, for TLS). * **On disk.** `node.key` (and `node.key.next` during a renewal) then holds only the record: a PEM block of type `BCD TPM NODE KEY` whose bytes are the unique, mode 0600. A key in the TPM does not load without that TPM. * **When.** Every new node key is made in the TPM when there is one: at `bcd join` and at every renewal's next key (under `[helper]`, in the helper's TPM connection). A host build uses `/dev/tpmrm0` when it exists; a fakehw build uses its software TPM only with `BCD_FAKEHW_TPM_KEYS=1`. When the TPM cannot hold the key, bcd logs a warning and makes a PKCS#8 file key instead, except at the renewal of a certificate that names an AK, which fails rather than leave the TPM (see [renewing a TPM-held key](#renewing-a-tpm-held-key)). A node that joined with a file key moves into the TPM at its next renewal. * **Under `[helper]`** (8870b8d, security review 3). The unprivileged `bcd node` never opens the TPM. The kernel's resource manager filters no command, so whoever can open `/dev/tpmrm0` can change the hierarchies' authorization, clear the TPM through its (empty) lockout authorization or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys) and attestation. `bcd join` and `bcd helper` run as root and open the TPM themselves; an unprivileged `bcd node` reaches its node keys only through the helper's `tpm.*` operations, which name a key by its 32-byte unique, and checks every signature the helper returns against the key's public half. The helper builds every TPM command itself, so the node picks no template, handle, hierarchy or authorization (see [node keys through the helper](/docs/daemon#node-keys-through-the-helper)). Install scripts from 0ba2a79 until 8870b8d put the `bcd` user in the `tss` group instead; an upgrade run takes it out again (see [running bcd node unprivileged](/docs/daemon/install#running-bcd-node-unprivileged)). * **What it protects.** Such a key cannot be copied off the machine, but anyone with the owner hierarchy's authorization on that TPM (root on the node) can derive and use it: non-exportable, not root-proof. * **Seeing it.** `bcd doctor` has a `node key` check: `ok` "in the TPM", `ok` for a file key on a host with no TPM, and `warn` for a file key on a host with a TPM ("the next renewal moves it into the TPM (through bcd helper when bcd node runs unprivileged)"). Under `[helper]` its `tpm` check (e6c3f53) fails while the node's user is still in the `tss` group. `bcd status` prints `node key in the TPM` or `node key a file` on its identity line, and its JSON carries `nodeKey` (`tpm` or `file`). What the control plane knows about it depends on how the key was made: | When | Proven to the control plane | Where it shows | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | A `tpm` join (cf24876) | Yes: the AK certifies the key (`CertifyNodeKey`, `TPM2_Certify` over `sha256(csr)`) and `tpmAttestor` checks it, refusing the join under the default `nodeKey: 'certified'` when it is missing | `nodeKey: 'tpm'` in `node.joined` and the `node.joined` or `node.reclaimed` audit row; `urn:bc:tpm-ak:` in the certificate (d94f1f3) | | A join with any other attestor | No: bcd still makes the key in the TPM when it can, but sends no certification | `nodeKey: 'unattested'`; no statement in the certificate | | A renewal of a certificate that names an AK (d94f1f3) | Yes: the same AK certifies the next key (`RenewRequest.key_attestation`), and JoinService refuses the renewal without it | the statement, carried into the new certificate and its `node_certs` row | | A renewal of any other certificate | No: the next key is made in the TPM when bcd can open one, but nothing proves it | nothing; the certificate and `node_certs` do not record where the key lives | So for a certified `tpm` join the proof lasts: every certificate the node holds from then on names a key that its TPM made and certified. For every other node the control plane has only bcd's word (`bcd status` and `bcd doctor` report it locally). What is tested, and with what TPM: | Test | TPM | What it proves | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TestTSContract` (`daemon/internal/tpm`) | `tpmsim` fixture, with a manufacturer EK certificate in NV (b973e05) | bcd's round-1 evidence is still byte for byte the fixture's, including the EK certificate bcd reads from NV index `0x01C00002` (`tpm.EKCertIndex`) and sends as `ekCert`, and the challenge `tpmAttestor` made for it activates and quotes through bcd's client (TypeScript to Go); since cf24876 the answer also carries the AK's certification of a node key held in the sim (`answer.nodeKey`) | | `tests/pki-attestors-tpm.test.ts` | `tpmProducer` over a `SoftwareTpm` whose `nodeKey` it certifies as bcd does | under `nodeKey: 'certified'`, a certified CSR key is admitted with `nodeKey: 'tpm'`; no certification is `node-key-not-certified`, and a certified key other than the CSR's is `node-key-not-csr`; `nodeKey: 'any'` admits the uncertified answer without a `nodeKey` claim | | `tests/pki-attestors-tpm-bcd.test.ts` | the same fixture | `tpmAttestor` with `ekRoots: [ek-root.der]` and no `allowUncertifiedEk` verifies the EK certificate's chain, challenges bcd's evidence and admits bcd's answer, with its node key certified (`nodeKey: 'tpm'`), while a CSR key other than the certified one is `node-key-not-csr`; it refuses the EK under another root (`ek-certificate-chain`), and the answer for another CSR (`challenge-for-another-csr`) or after the challenge expired (`challenge-expired`) (Go to TypeScript) | | `tests/e2e/bcd-tpm.test.ts` (`BCD_E2E=1`, WSL or Linux; db9bd46, 8a9af54) | `tpmsim` in a fakehw bcd, with an EK certificate from a manufacturer root the test makes (`BCD_FAKEHW_TPM_EK_CA`) and the node key in that TPM (`BCD_FAKEHW_TPM_KEYS=1`) | a real `bcd join --attestor tpm` with no token against the dev stack's JoinService, whose `tpmAttestor` trusts only that root (`ekRoots`, no `allowUncertifiedEk`): two rounds, `resolve` seeing the EK certificate chained to the manufacturer and the SHA-256 PCR bank 0 to 7, the node in the resolved pool and bound to `sha256(EK)` in `node_bindings`; `node.key` a `BCD TPM NODE KEY` record; the node key certified by the AK, so `node.joined` says `nodeKey: 'tpm'` for this join and for the rejoin below (cf24876); with its identity removed (a reinstall) the same TPM joins as the same node; `bcd node` on that identity, signing its session and statements in the TPM, is adopted; since d94f1f3 the certificate carries the AK's `urn:bc:tpm-ak:` statement, and `bcd node` renews (`BCD_RENEW_AFTER=3s`) with the AK certifying the next key, which stays a `BCD TPM NODE KEY` record while the new certificate keeps the statement | | `tests/server-join-tpm-renew.test.ts` (d94f1f3) | `tpmProducer` and `softwareTpmCertify` over `SoftwareTpm` | a certified join names the AK; a renewal without `key_attestation` is `key-attestation-missing`, another TPM's AK `key-attestation-ak`, a certification of another key `node-key-not-csr`, one for another CSR `node-key-nonce`; a good renewal keeps the statement although its CSR asks for no SAN, and the next renewal needs the AK again; an uncertified join (`nodeKey: 'any'`) gets no statement and renews without one | | `internal/agent` `TestRenewCertifiedByTheAK` (d94f1f3) | `tpmsim` | a certificate naming the sim's AK renews with the next key made in that TPM and certified by it; with no TPM the renewal fails (`unavailable`), and with another TPM (another AK) it fails before any request, never falling back to a file key | | `internal/helper` `TestHelperTPMKeys` (8870b8d, Linux) | `tpmsim` behind a helper server on a test socket | the key and AK the node gets through `tpm.*` are the same as the TPM's own; signing and certifying; at most 8 keys loaded (least recently used unloaded) and a released key derived again on use; refusals of bad shapes, unknown verbs and a helper without a TPM | `scripts/daemon-tpm-fixture.mjs` regenerates the fixtures in `daemon/internal/tpm/testdata` after a deliberate change of either side, so a format change fails a test rather than a machine's join. Since b973e05 the contract runs the attestor as against a real TPM: `ek-root.der` is a test manufacturer root and `ek-cert.der` the EK certificate it issued for the sim's EK (made once by the Go test, and kept with `sim.json` when the fixtures are regenerated), and the attestor verifies that chain through `ekRoots`. Since 8a9af54 the dev-stack e2e does the same: a fakehw bcd given `BCD_FAKEHW_TPM_EK_CA` (a PEM file with the root's certificate and its PKCS#8 key) issues its software TPM an EK certificate in NV at each load, so `allowUncertifiedEk` is no longer used by any bcd test. Every test uses a software TPM: no test has run against a hardware TPM, a real manufacturer's EK certificate or a real `/dev/tpmrm0`. ## Single-use evidence [#single-use-evidence] Tokens are not the only thing that must not be used twice. `cloud-iid` and `discovery` record what they accepted in `attestation_uses (attestor, subject, expires_at)` inside the join transaction: | Attestor | Subject | Kept until | | ----------- | ----------------------------------------- | ---------------------- | | `cloud-iid` | SHA-256 of provider, account and instance | forever | | `discovery` | SHA-256 of the signed assertion body | the assertion's expiry | A second join with the same subject is refused. `pruneAttestationUses(tx, { batch? })` from `@better-compute/postgres` deletes expired rows, 5,000 per call by default; see [retention](/docs/operations/retention). Single-use tokens (`provisioned`, `discovery`) must have exactly one use left, so a token minted for several machines is never accepted as a machine credential. ## Two-round joins [#two-round-joins] Some attestors cannot decide in one request: `tpm` must challenge the TPM, and `discovery` or `tpm` may wait for an operator. Such an attestor returns an `AttestationDeferral` instead of a result: ```ts interface AttestationDeferral { deferred: 'approval' | 'challenge'; requestId?: string; // approval: the join request retryAfterSeconds?: number; // approval: default 30 challenge?: Uint8Array; // challenge: attestor-specific bytes notAfter?: Date; // challenge: answers after this are refused } ``` JoinService then commits the transaction (so the attestor's own rows, such as a pending join request, persist), issues nothing and spends nothing, and answers with a signed `JoinResponseBody` whose `deferral` holds either `JoinPending{ request_id, retry_after_seconds }` or `JoinChallenge{ attestor, payload, not_after }`. The node id and chain are empty, and `csr_sha256` still binds the answer to the request. The agent re-sends `Join` with the **same CSR**: after the wait, or with its answer to the challenge in `attestation`. Attestors therefore consume tokens and record uses only on the final round. On the agent side, `openJoinResponse` reads a response as joined, pending or challenge, and `verifyJoinResponse` refuses a deferral. `runAgentJoin` in `@better-compute/testing` is the reference retry loop. ## Requiring an attestor [#requiring-an-attestor] By default a pool takes a machine whichever attestor admitted it. Since 1516728 a pool can require hardware attestation with `policy.joinAttestors`, the ids of the attestors that may admit a machine into it: ```ts title="tpm-pool.ts" const pool = await client.pools.get({ tenantId: 'root', poolId: 'pool_01k…' }); await client.pools.update({ tenantId: 'root', poolId: pool.id, // The policy is replaced as a whole, so keep the rest of it. policy: { ...pool.policy, joinAttestors: ['tpm'] }, // or ['tpm', 'x509pop'], or ['join-token', 'tpm'] to keep tokens }); ``` * **Where it is checked.** JoinService checks it in the transaction that resolves the pool, before any reclaim or insert. A join or a reclaim admitted by an attestor that is not listed is refused with `ACCESS_DENIED`, reason `pool-attestor`. A token join is `join-token`, so a pool that lists only `tpm` refuses tokens, and a token reclaim of a TPM-bound machine (`allowReclaim` with the same trusted hardware key) is refused while its TPM still reclaims it. * **Who sees the refusal.** The attestor has already verified the machine, so the refusal is attributed to the pool's owner: `security.join.refused` with the attestor and reason `join:pool-attestor`, and the token or node id when known. * **Validation.** `pools.create` and `pools.update` refuse (`INVALID_INPUT`) a value that is not an array of 1 to 16 ids of attestors this control plane runs: `join-token` and every attestor registered in `attestors` or by a module or plugin. * **Moving nodes.** No API moves a node between pools. A node gets its pool at join, where the check runs, or at `metal.enroll`, whose machine gets its identity only through JoinService. Tested by `tests/server-join-pool-attestors.test.ts`. ## Choosing an attestor [#choosing-an-attestor] | Situation | Use | | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | A server you install by hand or with configuration management | `join-token`, with `expect` when the firmware reports placeholder identifiers | | A node whose certificate expired while it was offline | `x509pop` | | The OS a Better Compute deploy or a provider bootstrap installed | `provisioned` | | A ramdisk during bare-metal commissioning | `discovery`, with `approval` if unexpected machines should wait for a person | | A VM your provider account created | `cloud-iid` | | Hardware identity that must survive reinstalls, on TPM 2.0 machines | `tpm` (v0.5), with `bcd join --attestor tpm`; add `policy.joinAttestors: ['tpm']` to refuse every other way in | To write your own attestor, see [attestors](/docs/extending/attestors).