Joining

Joining turns a machine into a node: JoinService checks the machine's evidence, creates or reclaims the node row, and signs a short-lived client certificate that the daemon uses for every later connection. This page covers joining with a join token, the v0.1 path for servers you already run (a brownfield adopt). Bare-metal machines join through other attestors; see attestation. A machine with a TPM 2.0 can join with no token at all, bcd join --attestor tpm (8545688; see bcd with its TPM), and a pool can refuse token joins altogether with policy.joinAttestors (see requiring an attestor).

Before you start

• A pool that runs the daemon. The node lands in one pool of one site. The pool must have agent: 'bcd'; joinTokens.create answers INVALID_STATE for a pool whose machines run no daemon. See pools.
• The machine PKI. better-compute pki init --trust-domain compute.example.com writes the CAs and the two JoinService keys. Pass them to the server as pki.join (pkiJoinCrypto({ trustDomain, nodeCa, signingKey, encryptionKey })); without them every join fails with INVALID_CONFIG. See PKI.
• A gateway the machine can reach. The daemon only dials out, to the regional gateway on port 443. See the gateway.

Join tokens

A join token is a single-purpose credential: it lets a limited number of machines join one pool, for a limited time. Its bearer form is bcjt_<id>.<secret>, where the id is jt_… and the secret is 256 random bits in base64url. The token is returned once, when it is created. The control plane stores only the SHA-256 of the secret, so a database dump cannot be replayed as tokens.

Creating a token

joinTokens.create is decided as compute:join-tokens:create on compute-pool/{poolId}. Only the pool's owner tenant can mint tokens for it; any other tenant gets NOT_FOUND.

const created = await compute.api.joinTokens.create(credential, {
  tenantId: rootTenantId,
  poolId,
  ttlSeconds: 3600,
  maxUses: 1,
  expect: { serial: 'SN-7701' },
  idempotencyKey: 'adopt-rack-b7-u12',
});

// `bcjt_jt_….…`: hand it to the machine out of band. It is never shown again.
console.log(created.token);

The same call from the CLI's join-tokens create (44915e5), which writes the token to a new 0600 file with --out instead of printing it:

better-compute join-tokens create --tenant $TENANT --pool $POOL --ttl 1h \
  --expect-serial SN-7701 --key adopt-rack-b7-u12 --out ./join-token
better-compute join-tokens list --tenant $TENANT --pool $POOL
better-compute join-tokens revoke jt_01k5w… --tenant $TENANT --pool $POOL

--ttl is a duration from 1m to 24h in whole seconds; --max-uses, --allow-reclaim and --expect-product-uuid, --expect-serial, --expect-mac and --expect-gpu-uuid map to the fields below. --out never writes over a file (ALREADY_EXISTS), and it refuses a replayed --key, whose answer carries no token (INVALID_STATE: revoke that token and create a new one). join-tokens list never shows a secret (--all adds revoked, expired and used-up tokens), and join-tokens revoke takes the token id and refuses a pasted token (anything with a .), so a secret is never sent as an id.

The answer is the token record (id, poolId, siteId, role: 'node', usesLeft, expect, allowReclaim, hosted, expiresAt) plus token. A retry with the same idempotency key returns the record without token; the same key with a different pool, expect, allowReclaim or hosted fails with CONFLICT. Creating a token publishes security.join-token.created, whose pinned field says whether it has an expect binding, with hosted when it is set.

A hosted token (--hosted kubernetes in the CLI) is for bcd as a DaemonSet on a cluster whose admins have root on its machines through privileged pods. A node that joins with it is tenant-controlled, with a node-tenant certificate that never receives platform secrets, unless the token's pool has policy.kubernetes.platformOperated: true and the platform owns that pool (259d82c): its owner is the owner of its site, the site is not custom, and, when the deployment names a root tenant, the owner is that tenant, so a tenant cannot vouch for its own cluster by editing its pool's policy. The token decides, never the machine's claims, and it only lowers platform trust (a tenant's own site stays tenant-owned); a reclaim through a hosted token lowers an existing platform node too.

joinTokens.list (compute:join-tokens:read) lists a pool's live tokens, and every token with includeInactive: true; pages hold up to 200 (100 by default). joinTokens.revoke (compute:join-tokens:revoke) ends a token at once and publishes security.join-token.revoked. Revoking twice is not an error.

Token roles

Every token row has a role, which decides what kind of agent it can admit.

A token presented for another role is refused. Modules mint the last two through ctx.pki.issueJoinToken, and they travel through the control plane only as secret references, never in a command body. They are covered in attestation.

Binding a token to one machine

A token with maxUses: 1 still admits whichever machine presents it first. expect pins it to one machine: every identifier you name must be among the machine's claims. The comparison normalizes formats first:

• productUuid is compared lower case and dashed, and a placeholder UUID never matches;
• serial matches the product, board or chassis serial, case-insensitively;
• mac must be a permanent, globally administered MAC of a physical NIC (separators and case do not matter);
• gpuUuid is compared case-insensitively.

A mismatch refuses the join without consuming the token.

The join flow

1. Key and CSR. The daemon generates a P-256 key and a PKCS#10 request. bcd makes the key in the host's TPM when it can open one, and its 0600 node.key under the data directory then holds only the key's record; otherwise node.key is the key itself, a PKCS#8 file (see the node key in the TPM). Only a tpm join proves where the key lives: there the key is made in the attestor's TPM and the TPM's attestation key certifies it (cf24876; see tpm).
2. Sealed token. The token is encrypted as a compact JWE (ECDH-ES+A256KW, A256GCM) to the JoinService encryption key pinned in the daemon's config. The protected header carries typ: bcjt+jwe, iat and csr, the SHA-256 of the request. The gateway relays the call but can neither read the token nor replay the sealed blob with a CSR of its own: JoinService refuses a seal made for another CSR.
3. Join. Join carries the sealed token, the CSR, the hardware claims (product UUID and serials, permanent MACs, GPU UUIDs, hostname), the attestor name (empty means join-token) and the requested role. It travels over TLS that authenticates the gateway against the pinned serving CA.
4. Admission, in one transaction. JoinService opens the seal, then the join-token attestor locks the token row and checks the secret (in constant time), revocation, expiry at compute_now(), the remaining uses and the role, and consumes one use. JoinService then enforces expect, the placeholder rule and the reclaim rules below, checks that the pool runs the daemon and belongs to the token's tenant, and inserts the node in adopting with identityGeneration 1. It writes the node's authorization projection row, publishes node.joined and writes the node.joined audit row. Both carry the attestor, whether the join was a reclaim (the audit action is then node.reclaimed), hardwareKeyTrusted, the certificate serial and nodeKey: tpm when the attestor proved the node key lives in the machine's TPM, else unattested, which is every token join (cf24876). A node in a pool without acceptance checks (and a site agent) is queued for its controllers at once and adopted; when the pool runs acceptance checks the node is still disconnected, so JoinService does not queue it, and its first Hello wakes the controller that starts acceptance.
5. Signed response. The answer is JoinResponse{ body, sig }, signed with the pinned join-signing key. The body holds the node id, the certificate chain, the command-signing trust bundle, csr_sha256, the trust domain, the site and the role. The daemon checks, in order: the signature under the pinned key, that csr_sha256 is its own request's, that the leaf certifies its key, the SPIFFE SAN, and the role.
6. Session. The daemon opens its session with Hello. See sessions.

No step of the join calls better-iam.

Refusals

The agent never learns why a join failed: every refusal is the same ACCESS_DENIED ("Join refused"), so a stolen or guessed token reveals nothing. The reason is recorded server-side as a security.join.refused event with attestor, reason (<attestor>:<reason>), role, and hashes only (csrSha256, hardwareKey). The event goes to the token's tenant only once the token's secret verified; before that it goes to the root tenant, so an unauthenticated caller cannot write into another tenant's event stream. Events are rate-limited to 60 per attestor and reason per minute, and each carries suppressed, the count dropped since the last one. The reasons you will see for token joins:

Two failures are not refusals: a missing JoinService configuration is INVALID_CONFIG, and two joins of the same machine racing each other end with one CONFLICT ("This machine joined concurrently; retry").

Node certificates

The certificate is the node's identity from here on. Its properties:

• Key and lifetime. ECDSA P-256, at most 24 hours (JoinService's certTtlSeconds, default and maximum 86,400). Ramdisk certificates last at most 2 hours, and an attestor may shorten either.
• Name. Exactly one SPIFFE URI SAN, the SPIFFE ID spiffe://<trustDomain>/site/<siteId>/node/<nodeId>, first; any further URI SANs are urn:bc: statements about the node key, never identities. Today the only one is urn:bc:tpm-ak:<nameAlg>:<digest>, written after a certified tpm join (d94f1f3). Names and statements come only from JoinService, never from the CSR, and no tenant appears in the certificate.
• Role. One subject attribute OU=bc-role:<role> holds node, site, ramdisk or node-tenant. The certificate carries no private-arc OID, so Go's crypto/x509 (and bcd) can parse it. See PKI.
• Record. Every issued certificate is a node_certs row with its serial, identity generation, SPKI hash and DER, which is what later renewals and attested results are checked against.
• Allowed kinds. An attestor may restrict the command kinds a certificate accepts (allowedKinds: exact kinds or prefix.*). The command queue refuses other kinds at enqueue with INVALID_STATE, and the gateway checks again at dispatch. See commands.

Renewal

bcd renews at about 60% of the lifetime with Renew, proving possession of its current key. The request carries a new CSR, the current chain, client_time and pop_sig: a signature by the current key under purpose renew-pop over the CSR's SHA-256, the node id and the client time. JoinService in api verifies it; the gateway only relays, so a compromised gateway cannot mint certificates. A renewal is refused unless:

• the presented leaf is byte for byte a node_certs row, unrevoked, unexpired and of the node's current identity generation;
• the node is not retired, and the caller's mTLS identity is that node;
• client_time is within 60 seconds of server time, and the same proof was not used in the last 120 seconds;
• when the leaf names a TPM attestation key (urn:bc:tpm-ak:), the request carries key_attestation: that AK's TPM2_Certify of the CSR's key over sha256(csr), as a key the TPM made and never lets out (d94f1f3).

The new certificate keeps the role, the allowed kinds and the urn:bc: statements of the old one. JoinService reads the statements from the presented leaf, which it has just matched to its own node_certs row, never from the CSR, so a node can neither drop the AK statement nor renew with a key outside its TPM: a missing or failing attestation is refused (key-attestation-missing, key-attestation-ak, node-key-*), never downgraded. bcd builds the attestation itself and fails the renewal loudly, without sending it, when the TPM is unavailable or derives another AK. See renewing a TPM-held key for the checks and for how such a node recovers by joining again. A certificate without a statement (every join other than a certified tpm join) renews with a proof of possession only: bcd still makes the next key in the TPM when it can, but nothing proves it to the control plane.

Every refusal answers the agent with the same ACCESS_DENIED ("Renew refused"). Since 0e42018 JoinService also publishes security.renew.refused with the reason, to the node owner once the caller proved the current key and to the root tenant otherwise; see refused renewals.

A certificate that expired while the machine was offline cannot be renewed; when the server has the x509pop attestor configured, the machine re-joins with it within 7 days instead (see attestation). bcd keeps such an identity: it still loads, so bcd join answers already joined, and bcd doctor fails its identity check with "the certificate expired at ...; join again as root: bcd join --reset --yes, then bcd join". Since a1c1a46 that is the way back: --reset --yes moves the old identity aside and the join that follows presents a new token (see joining again with --reset):

# as root; the token from standard input, as for any join
bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt

Adopting a brownfield server

BETTER_COMPUTE_SECRET=... better-compute pki init --trust-domain compute.example.com
better-compute pki issue-gateway --san gw.eu.example.com

better-compute daemon install-script --pool $POOL --site $SITE \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=<hex>,arm64=<hex> > install-bcd.sh

BCD_JOIN_TOKEN=bcjt_... sudo -E sh install-bcd.sh
# or, keeping the token out of the environment:
sudo sh install-bcd.sh < token-file

The install script:

• reads the token only from BCD_JOIN_TOKEN or standard input, never from arguments or a URL, and refuses anything that does not look like bcjt_…;
• downloads bcd for the machine's architecture from {base-url}{base-path}/downloads/bcd/{version}/linux-{arch} and checks its SHA-256;
• installs these files, then runs bcd join with the token on standard input and enables bcd.service.

gateway_url = "https://gw.eu.example.com"
site = "site_…"
serving_ca_spki_pin = "sha256/…"
join_signing_key_pin = "sha256/…"
join_encryption_key_pin = "sha256/…"

The token is consumed by the join and never written to disk. Run without a token on a host that already has /etc/bcd/config.toml, the script upgrades bcd in place and restarts it.

The join key and its fallback

The pins alone cannot seal a token, so the script also writes the JoinService encryption key to /etc/bcd/join-encryption.jwk. bcd join uses it only when its SPKI hash equals join_encryption_key_pin. When the file is missing or the key was rotated, the daemon fetches GET {basePath}/.well-known/better-compute, whose join section the server publishes when pki.join is configured:

{
  "name": "better-compute",
  "version": "0.1.0",
  "basePath": "/api/compute",
  "protocol": { "min": 1, "max": 1 },
  "join": {
    "encryptionJwk": { "kty": "EC", "crv": "P-256", "x": "…", "y": "…" },
    "encryptionKeyPin": "sha256/…",
    "signingCertificate": "MIIB…",
    "signingKeyPin": "sha256/…"
  }
}

The daemon accepts these keys only when they match its pins, so the fetch needs no trust in the transport.

bcd implements half of it. Its configuration has no control-plane URL, so the fetch happens only when join-encryption.jwk is absent and bcd join --server https://… names the control plane: bcd then reads <server>/.well-known/better-compute and uses join.encryptionJwk if it matches join_encryption_key_pin. That is how an OS a deploy installed joins (--server, --token-file and --node-id, from the deploy's user data), and how bcd as a DaemonSet joins a cluster's nodes. --token-file takes only a file its owner alone can read (0600 or 0400; a Kubernetes Secret needs defaultMode: 0400), a regular file of at most 4 KiB, and deletes it after the join; a read-only Secret mount it cannot delete is a warning, not a failure (c29bb16). A key file that is present but does not match the pin still fails bcd join: after rotating the key, generate the install script again and rerun it, which rewrites the file and the pins.

When there is no config.toml at all, bcd join --server goes further and builds its configuration from the document: the gateway from its gateway section, which the server publishes when its options set gateway: { url, serverName? } (with servingCaPin computed from pki.servingCa). The pins do not come from the document: a host build requires --serving-ca-pin, --join-signing-pin and --join-encryption-pin from the install and refuses a document whose pins differ. It joins, then writes config.toml and join-encryption.jwk. A provider server installed with a token, the control plane's URL and the pins (from bootstrapPins(bundle)) joins this way; see joining from the well-known document.

{
  "gateway": { "url": "https://gw.eu.example.com", "serverName": "gw.eu.example.com", "servingCaPin": "sha256/…" }
}

Placeholder hardware and reclaims

JoinService derives the machine's hardware key from its claims, exactly as the Inventory controller does from inventory. Firmware often reports placeholder values instead of real identifiers (To Be Filled By O.E.M., Default string, System Serial Number, all-zero UUIDs, one repeated character). They are dropped before hashing, and the key is trusted only when at least two independent identifiers remain (product UUID, a serial, permanent MACs, GPU UUIDs).

• Placeholder hardware. When the key is missing or untrusted, or neither a product UUID nor a serial survived, a join-token join is refused (placeholder-hardware) unless the token has an expect binding. With one, the node joins with an untrusted key. Machines like this can also be admitted through operator approval with the discovery or tpm attestors (see below).
• Reclaims. When a machine with a trusted key that already belongs to a node joins with a new key (a reinstall, a replaced disk), the join is a reclaim. It needs a token with allowReclaim, in the same tenant, site and pool, for the same role; otherwise it is refused (reclaim). The node keeps its id, its identity generation goes up by one, every older certificate is revoked (reclaimed, with a security.certificate-revoked event), acceptance runs again, and the audit row says node.reclaimed. A machine with fewer than two identifiers never reclaims: it joins as a new node. In a pool whose policy.joinAttestors does not list join-token, a token join or reclaim is refused (pool-attestor, 1516728).

Join requests and approvals

Some joins need a person to decide: a PXE-booted machine nobody expected, or a TPM whose endorsement key is not on any allow-list. The attestors that defer (discovery with approval: 'unknown-machines' or 'always', and tpm with enrollment) record a join request and answer the daemon with a signed deferral instead of a certificate. Nothing is issued and no token is spent.

The request records the machine (keyed by its trusted hardware key or TPM endorsement key hash, else by site and MAC), the claims, and the SHA-256 of the CSR the retry must present. A machine has at most one pending request; a newer attempt refreshes it with its new CSR. An approval admits exactly one join: the retry with that CSR consumes it inside its own join transaction. A rejected request makes a retry with that CSR fail.

The joinRequests group (v0.2) is decided on the pool, like join tokens:

const { items } = await client.joinRequests.list({ tenantId, poolId });
for (const request of items) {
  console.log(request.id, request.attestor, request.mac, request.vouchedBy, request.claims);
}

await client.joinRequests.approve({
  tenantId,
  poolId,
  requestId: items[0].id,
  ttlSeconds: 900,
  reason: 'rack B7 intake, ticket 4411',
});

From the command line (573b0b0), where approve --ttl is in seconds (60 to 86,400):

better-compute join-requests list --tenant $TENANT --pool $POOL            # pending by default; --state all
better-compute join-requests approve jreq_01k5w… --tenant $TENANT --pool $POOL --ttl 900 \
  --reason 'rack B7 intake, ticket 4411'
better-compute join-requests reject jreq_01k5w… --tenant $TENANT --pool $POOL --reason 'unknown serial'

A request shows what the attestor saw: attestor, claims, hardwareKey (hex), mac, ip, workflowId, vouchedBy and bootLeaseEpoch (the site agent that vouched for a discovered machine), csrSha256, and the decision fields decidedAt, decidedBy, reason and consumedAt. Decisions publish security.join-request.approved and security.join-request.rejected.

Troubleshooting