Node identity

A node's identity is a P-256 key and a short-lived certificate the control plane's JoinService issued for it. bcd creates it once with bcd join, replaces the key and certificate at 60% of every certificate's lifetime, and uses the key for every TLS handshake and every statement the node signs. This page describes the node's side: the files, the join, renewal, revocation, recovery with bcd join --reset, and node keys held in the host's TPM.

The control plane's side is elsewhere: joining nodes covers tokens, the join transaction and certificates, attestation covers each attestor and the TPM proofs, and PKI covers the keys and pins behind it all.

What the identity is

Everything lives in the data directory (data_dir, default /var/lib/bcd):

identity.json holds:

The identity outlives a lost journal: a corrupt or wiped journal starts over with a new journal id, but the node keeps its key and certificate (see journal loss).

Loading it

Every agent mode loads the identity with LoadState at start:

1. No identity.json is this node has not joined (run bcd join).
2. A node.key.next whose public key the stored leaf certifies is promoted to node.key (a renewal stopped after it stored the certificate); any other node.key.next is removed (a renewal stopped before).
3. node.key must be the leaf's key. If it is not: the node key does not match the node certificate; join again: bcd join --reset --yes, then bcd join. For a TPM key the message adds the TPM derives another key from node.key (was the TPM cleared?), since a cleared owner hierarchy derives another key from the same record.

bcd doctor's identity check reports the same: a warning when the node has not joined, a failure for an identity that does not load or an expired certificate, each ending with the way back.

Joining

bcd join creates the identity. It runs as root, takes the join credential from one source, and never from a command-line argument (a positional argument is refused without being echoed).

--role site joins a site agent with a site token; the token decides what is allowed. --server, the three pin flags and --data-dir bootstrap a machine that has no config.toml from the control plane's well-known document (see joining from the well-known document). bcd produces no x509pop or cloud-iid evidence: those attestors exist in @better-compute/pki, but a bcd node whose certificate expired offline re-joins with a token, the TPM or --reset instead.

What the join does

agent.Join (daemon/internal/agent/join.go):

1. The key. For a tpm join on a host whose node keys use the TPM, the key is made in the attestor's own TPM connection so the AK can certify it; otherwise a key in the host's TPM when there is one, else a PKCS#8 file key (see node keys in the TPM). Then a PKCS#10 CSR.
2. The seal. A token is sealed to the JoinService encryption key, which must match join_encryption_key_pin (from join-encryption.jwk beside the config, or the well-known document with --server when that file is absent). The seal is bound to the CSR's SHA-256.
3. Join. Sent over TLS pinned to the serving CA (serving_ca_spki_pin), with the CSR, the hardware claims (product UUID and serials, permanent MACs, GPU UUIDs, hostname, boot id), the attestor, its evidence, the daemon version and the role. Retryable failures back off, up to 20 attempts; a gateway's grpc-retry-pushback-ms is waited out exactly.
4. The answer. pki.OpenJoinResponse verifies the signature under join_signing_key_pin, that csr_sha256 is this request's, that the leaf certifies this key, the SPIFFE ID and the role. A deferral is answered with the same CSR: a JoinChallenge for this attestor (at most 3 rounds, refused once its not_after passed), or a JoinPending approval, waited out (at least one second) and, for an attestor with a challenge, followed by a fresh round.
5. Root's record. On a node with [helper], when bcd join runs as root, the verified JoinResponse is written as signed to /var/lib/bcd-hostconfig/join-response.pb (0600) before the identity (a966cd4). A failed record fails the join and leaves no identity. bcd helper takes the node id and the command-signing roots for host units from it.
6. Persist. If the data directory held another node's identity, its journal.db is moved aside as journal.db.<oldNode>-<unix time>. A stale node.key.next is removed, then node.key and identity.json are written.
7. Hand over. Under [helper], root Lchowns identity.json and node.key to the [helper] user, so the unprivileged bcd node can read them (b4fcc2e); a name swapped for a symlink changes only the link.

It prints joined as <node> (site <site>, role <role>); certificate valid until <time>. On a machine whose identity loads, bcd join reads no credential, answers already joined as <node> (site <site>, role <role>) and exits 0, so a join unit can run on every boot; a different --node-id is refused. Every refusal from the control plane is the same ACCESS_DENIED ("Join refused"); the reason is only server-side (see refusals).

The pins

The install script writes them from the PKI bundle; a malformed pin stops bcd from starting.

Renewal

A node certificate lasts at most 24 hours (a ramdisk's at most 2). The agent renews at 60% of the leaf's lifetime (renewFraction), with Renew on the gateway, which relays to JoinService. renewOnce:

1. Chooses the next key. If the leaf names a TPM AK (urn:bc:tpm-ak:, written after a certified tpm join; d94f1f3), bcd loads that AK alone, checks its TPM name is the one the certificate names, and makes the next key in that TPM. Otherwise the next key goes into the TPM when the host has one, so a file-key node moves into the TPM here, else it is a file.
2. Writes node.key.next first, then builds the request: a new CSR, the current chain, the client time and a proof of possession by the current key (purpose renew-pop).
3. Attaches the AK's proof for a certificate that names one: key_attestation with the AK's public area, the new key's public area, and the AK's TPM2_Certify of it over sha256(csr).
4. Checks the answer: the new leaf names the same node, site, trust domain and role, and certifies the new key.
5. Stores in order: node.key.next again (in case something removed it), then identity.json, then renames node.key.next over node.key. A crash at any point leaves a key and certificate that match. The new state keeps the trust version, roots, bundle and server time. The old key is unloaded 5 minutes later, once nothing can still be signing with it.

A failed renewal is logged as renewal failed with the attempt and expiresIn, and retried with backoff from 30 seconds to 10 minutes, or exactly when a busy gateway says (it allows 6 renewals per node and hour). There is no downgrade for a certificate that names an AK: when the TPM is unavailable, or derives another AK, the renewal fails before anything is sent, and keeps failing until the certificate expires.

The control plane's checks and refusal reasons are on renewal and renewing a TPM-held key. A certificate that expired cannot be renewed; see joining again.

Revocation

The control plane revokes a node's certificates with nodes.revokeIdentity, by deleting the node, or on its own for a certificate that looks cloned (see revoking a node's identity). A gateway with a revocation feed then sends the node's live session a RevocationNotice and closes it.

On the node, the notice is logged as certificate revoked with its reason, the agent stops (ErrRevoked), and bcd node exits 1 with agent: this node's certificate was revoked. It does not redial with that certificate, and nothing on the node deletes or renames the identity. The revoked certificate cannot renew, re-join or open a session again (the gateway answers UNAUTHENTICATED), and a later identity generation means older certificates stay dead. The machine comes back only by joining again with fresh evidence: a new token, or the TPM, after bcd join --reset.

Joining again with --reset

An identity that loads but no longer works (expired, revoked, refused at renewal, naming an AK this TPM no longer has) makes bcd join answer already joined forever. Since a1c1a46, --reset moves it aside first:

# as root: show what would move (moves nothing, exits 2)
bcd join --config /etc/bcd/config.toml --reset
# move it aside and join again, with a new token on standard input
bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt
# or with the TPM
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm

Whether the machine comes back as the same node depends on the attestor. tpm with the same EK and provisioned reclaim their node; a token join of a known machine needs a token with allowReclaim, or the old node deleted first. The cases are on joining again with --reset and recovering a node whose TPM changed. Under [helper], restart bcd-helper.service and bcd.service afterwards; the join also rewrites root's join record, so the helper binds host units to the new node.

Node keys in the TPM

bcd keeps the node key in the host's TPM when there is one (965c14d, 50e18f9, 0ba2a79). Whatever holds it, the key is a Go crypto.Signer, and nothing on the wire changes: the same P1363 signatures and the same kid.

internal/tpm is bcd's own TPM 2.0 client (f61b32d): it builds the TPM's command format directly over the transport, with no library, and sends only CreatePrimary, ReadPublic, NV_ReadPublic, NV_Read, StartAuthSession, PolicySecret, ActivateCredential, Quote, PCR_Read, Sign, Certify and FlushContext. The same package builds the tpm attestor's evidence (EK, AK, the EK certificate from NV index 0x01C00002) and its round-2 answer; see bcd with its TPM.

Through bcd helper

The kernel's resource manager filters no command, so whoever can open /dev/tpmrm0 could change the hierarchies' authorization, clear the TPM or trip its lockout. Since 8870b8d (security review 3) an unprivileged bcd node never opens it:

The node sends tpm.nodekey, tpm.sign, tpm.ak, tpm.certify and tpm.release, each naming a key by its unique, with a 30-second timeout. It checks every answer: the derived key must have the node-key attributes, every signature must verify against the key's public half (the helper's signature is not the node key's otherwise), and a certification must come from the AK the helper announced (the helper certified with another AK). What the helper checks is on bcd helper. The install script no longer puts bcd in the tss group, and bcd doctor's tpm check fails while it is still a member.

Seeing where the key is

What the tests cover