Installing bcd

A server becomes a node in two moves: an operator generates an install script once per pool, then runs it on each machine with a single-use join token. The script downloads a pinned bcd binary, checks its digest, writes a small configuration, installs a hardened systemd unit, joins the fleet and starts the daemon. The script contains no secret, so you can keep it in a provisioning repository; the token travels separately.

Serving the binaries

The api serves the daemon binaries itself when you configure downloads:

betterCompute({
  // …
  downloads: { dir: '/srv/bcd' }, // or { blob: blobStore }, or both (the blob store is tried first)
});

GET {basePath}/downloads/bcd/{version}/{os}-{arch} answers goreleaser's bare binary bcd_{version}_{os}_{arch}, read from downloads.blob under the key bcd/{version}/{file} or from {dir}/{file}. The route is public: the install script and self-update check the sha256 from the verified release manifest, not the transport. Responses are application/octet-stream with cache-control: public, max-age=31536000, immutable. The path must match a strict pattern (a semantic version with an optional leading v, linux, amd64 or arm64), so nothing else on disk can be reached; anything else, or a binary the deployment does not have, is 404. downloads needs dir or blob, or betterCompute() throws INVALID_CONFIG.

The same route serves the bcx-ipmi plugin (97b2ac2): GET {basePath}/downloads/bcx-ipmi/{version}/{os}-{arch} answers goreleaser's archive bcx-ipmi_{version}_{os}_{arch}.tar.gz, from the blob key bcx-ipmi/{version}/{file} or {dir}/{file}. The products are a fixed list, bcd and bcx-ipmi; any other name is 404. The route stays public because the archive's digest is in the same signed release manifest, and the install script checks the download against it (installing plugins from the release).

Generate the script

daemon install-script reads the public half of the PKI bundle for the pins, so it needs no secret. Create the bundle first with better-compute pki init (see PKI).

better-compute daemon install-script \
  --pool pool_h100 --site fsn1 \
  --gateway-url https://gw.eu.example.com \
  --base-url https://compute.example.com \
  --bcd-version 0.1.0 \
  --sha256 amd64=<hex>,arm64=<hex> > install-bcd.sh

Give exactly one of --sha256 and --release-manifest. With a manifest, the CLI verifies its signatures under the pins and threshold, checks that it is for product bcd and for --bcd-version (RELEASE_MISMATCH otherwise), and pins the one bare linux-amd64 and linux-arm64 binary it lists; archives, checksums and metadata files are skipped. See rollouts for signing releases.

Every value embedded in the script is checked against a narrow pattern before it is written, so nothing a caller passes can break out of the shell or TOML quoting:

A bad value fails with INVALID_INPUT naming the flag.

What the script does

The output is a POSIX sh script that runs with set -eu and umask 077.

Download the script before running it. Piping it into sh would leave no standard input for the token.

BCD_JOIN_TOKEN=bcjt_... sudo -E sh install-bcd.sh
# or, keeping the token out of the environment:
sudo sh install-bcd.sh < token-file

Joining with the TPM

A script generated with --attestor tpm (a006d9f) joins the machine with its TPM, through the control plane's tpm attestor, and carries no token step. Its header says the attestor's policy decides where the machine joins. It differs in two places:

• Instead of reading a token, it unsets BCD_JOIN_TOKEN and stops unless /dev/tpmrm0, the kernel's TPM resource manager, exists.
• Instead of the token join, it runs bcd join --config /etc/bcd/config.toml --attestor tpm < /dev/null, which returns at once on a machine that already joined, so the same script installs and upgrades. Then it enables and restarts bcd-helper.service (in helper mode, after chown -R bcd:bcd /var/lib/bcd) and bcd.service. As root on a node with [helper], the join also keeps the signed join record the helper binds host units to (see signed specs).

The join makes the node key in the same TPM and sends the TPM's certification of it, which tpmAttestor requires by default (nodeKey: 'certified', cf24876). The join runs as root and opens the TPM itself, with or without [helper]; under [helper] the unprivileged bcd node then reaches that key only through bcd helper (8870b8d), and never opens /dev/tpmrm0 (below). If the TPM cannot make the key, the join stops with an error. See bcd with its TPM.

better-compute daemon install-script --pool pool_tpm --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=<hex> --attestor tpm > install-bcd.sh
sudo sh install-bcd.sh

The control plane must run tpmAttestor with a resolve allow-list or enrollment for the machine's endorsement key, else the join is refused or waits for approval; --pool only names the pool in the header. A pool can then refuse token joins with policy.joinAttestors: ['tpm'] (see requiring an attestor).

Files on the host

The script writes the binary, config.toml, join-encryption.jwk, the unit and, on a plugin node, the drop-in and the (empty) plugin directory. The other paths belong to the daemon: bcd join writes the key and identity.json under data_dir, bcd node opens the journal there, serves the status socket at socket_path, and writes CDI specs for hold bindings (with env files under /run/bcd/allocations). On a host with a TPM (/dev/tpmrm0) the key is made in the TPM and node.key holds only its record, a PEM block BCD TPM NODE KEY; otherwise it is a PKCS#8 file (see the node key in the TPM). A renewal writes the next key as node.key.next and promotes it only once the new certificate is stored. A journal that does not open is moved aside as journal.db.corrupt-<unix time> and a new one starts. The token is consumed by the join and never written anywhere. With --proxy-ca-file the script also writes /etc/bcd/websocket-proxy-ca.pem. After its first self-update bcd owns <data_dir>/bin/bcd-a, bcd-b, current, previous and slots.json, upgrade.json while an update awaits its Welcome, update.json (the release floor and failed versions), and, under systemd, bcd-rollback.service and bcd.service.d/50-bcd-rollback.conf; /usr/local/bin/bcd then links to current. Under bcd helper the slots and both records are in root's [update] slots_dir instead (default /usr/local/lib/bcd), and the node downloads into <data_dir>/staging. A quorum node keeps batch-leaves.json and unbatched-destructive.json, and a site agent bmc-hosts.json, under data_dir.

config.toml

# Written by the better-compute install script. bcd refuses unknown keys.
gateway_url = "https://gw.eu.example.com"
site = "fsn1"
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."

bcd reads the file with daemon/internal/config, a strict subset of TOML 1.0: bare keys, basic or literal strings, decimal integers (with _ between digits) and booleans, in the top level or a bare [section]. Arrays, inline tables, floats, dates, multi-line strings, dotted or quoted keys, unknown keys, unknown sections, duplicate keys and wrong types are all errors with a line number, so a typo fails loudly. Files over 64 KiB are refused.

A pin is sha256/ plus the standard base64 of the SHA-256 of a SubjectPublicKeyInfo, the form HPKP and curl --pinnedpubkey use. bcd doctor warns when the file is readable or writable by anyone but its owner.

join-encryption.jwk

The pins alone cannot seal a token, so the script also writes the JoinService encryption public key as a JWK. bcd join uses it only when its SPKI hash equals join_encryption_key_pin. When the file is absent and bcd join --server https://… names the control plane, bcd fetches <server>/.well-known/better-compute (system roots, a 30-second timeout, at most 64 KiB) and uses its join.encryptionJwk, again only when it matches the pin. The design's fallback for a stale file is not implemented: a mismatched file, or a missing one without --server, fails bcd join with exit code 1. See the join key and its fallback.

An OS that deploy.image installed needs neither the script nor this file: the ramdisk writes its own pinned config.toml (gateway, server name, site and the three pins) and join-encryption.jwk into the installed root, both mode 0600, and the deploy's user data runs bcd join --server … --token-file /etc/bcd/join-token --node-id … (with the three pin flags when provision is given userData.pins), then bcd node. See profiles and images.

Joining from the well-known document

A machine with no config.toml at all, such as an OS that Hetzner Robot's bcd-join.service or provision's user data set up with a token, the control plane's URL and three pins, can still join. bcd join --server https://… --serving-ca-pin … --join-signing-pin … --join-encryption-pin … reads <server>/.well-known/better-compute for what the pins cannot say (where the gateway is), checks the rest against the pins, joins, and writes its own configuration.

The pins are sha256/<base64>, as in config.toml, and a host build requires all three to bootstrap ("joining without config.toml needs --serving-ca-pin, --join-signing-pin and --join-encryption-pin from the install"). A document whose published pin differs from one given is refused ("refusing to join"). Only a fakehw development build may join with no pins at all, taking them from the document (and it also accepts an http:// server). On a machine that already has config.toml, the configuration wins, and any pin flag given must match it.

The control plane publishes the gateway section when its options set gateway: { url, serverName? }, with servingCaPin computed from pki.servingCa; the examples read BETTER_COMPUTE_GATEWAY_URL and BETTER_COMPUTE_GATEWAY_SERVER_NAME. bcd fetches the document over the control plane's own TLS (system roots, a 30-second timeout, at most 64 KiB, redirects refused), refuses a document without gateway.url ("install /etc/bcd/config.toml instead"), joins, and only then writes config.toml (0600, with the site the join answered) and join-encryption.jwk (0644) beside it. Re-running it on a node that already joined exits 0 without a token, so a join unit can run on every boot. The pins come from the sealed PKI bundle: bootstrapPins(bundle) in better-compute/pki returns all three, and provision's user data (provisionModule({ userData: { pins } })) and a Robot bootstrap (vps({ bootstrap: { pins } })) render them into the machine; see profiles and images.

The systemd unit

By default the script installs bcd node running as root, so the unit narrows what root can do. With --unprivileged the same unit gets a drop-in that runs it as the bcd user beside a root bcd helper; see running bcd node unprivileged.

[Unit]
Description=better-compute node agent (bcd)
Documentation=https://github.com/better-compute/better-compute
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=10

[Service]
Type=simple
ExecStart=/usr/local/bin/bcd node --config /etc/bcd/config.toml
Restart=always
RestartSec=5s
UMask=0077
StateDirectory=bcd
StateDirectoryMode=0700
RuntimeDirectory=bcd
RuntimeDirectoryMode=0700
ConfigurationDirectory=bcd
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=-/etc/cdi -/var/run/cdi
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
LockPersonality=yes
SystemCallArchitectures=native
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
CapabilityBoundingSet=CAP_SYS_BOOT CAP_SYS_ADMIN CAP_SYS_RAWIO CAP_DAC_READ_SEARCH CAP_NET_ADMIN CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE

[Install]
WantedBy=multi-user.target

bcd node opens no TCP listener; everything it does starts with an outbound connection to the gateway. See privilege separation for the helper mode.

The plugin drop-in

The unit above cannot run sandboxed bcx plugins: its CapabilityBoundingSet has no CAP_SETUID or CAP_SETGID, which bcd needs to start each plugin under its own uid, and ProtectControlGroups=yes makes the cgroup tree read-only. The unit stays that tight; a node that runs plugins gets a drop-in instead. The script installs /etc/systemd/system/bcd.service.d/plugins.conf (0644) exactly when /etc/bcd/config.toml has a [plugins] section, and removes it otherwise, on every run, first install and upgrade alike:

[Service]
# bcx plugins ([plugins] in /etc/bcd/config.toml): their own uids, their own cgroups.
CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_KILL
Delegate=yes
DelegateSubgroup=main
ProtectControlGroups=no
# plugins.privileged in /etc/bcd/config.toml: runtime plugins that run VMs as root (Kata).
CapabilityBoundingSet=CAP_MKNOD CAP_SYS_RESOURCE CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_RAW CAP_SYS_CHROOT CAP_FSETID CAP_SETPCAP CAP_SYS_PTRACE
RestrictNamespaces=no
RestrictAddressFamilies=AF_VSOCK AF_PACKET
ProtectKernelModules=no

The last four lines are added only when [plugins] privileged names a plugin (3b7ae79, following d902063): a runtime plugin such as Kata runs VMs as root, so it needs close to root's capabilities (locking VM memory, device nodes, virtiofsd's ptrace), namespaces and mounts, vsock and packet sockets, and the vhost and vsock kernel modules:

Confined plugins are unchanged: each drops every capability when bcd switches it to its own uid, with no_new_privs and the seccomp filter. A node without privileged plugins gets none of these four lines.

--plugins makes a node a plugin node. The script then creates /usr/local/lib/bcd/plugins (root-owned, 0755, so each plugin's uid can reach its binary) and, unless the configuration already has a [plugins] section, appends one:

[plugins]
dir = "/usr/local/lib/bcd/plugins"
cgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins"
privileged = "kata"

The cgroup is a child of bcd's delegated cgroup. At start bcd creates it and enables the memory and pids controllers in it and in its parent; on systemd older than 254, which ignores DelegateSubgroup=main, bcd first moves itself into a main leaf, since cgroup v2 gives no controllers to the children of a cgroup that holds processes itself (d902063). If the cgroup cannot be prepared, no plugin runs and the log says why. --privileged-plugins NAMES implies --plugins and writes privileged; each name must be letters, digits, ., _ and -. Other plugins you put in the directory yourself, the binaries and their lines in its pins file (see driver plugins in bcd); the plugins a release ships the script can install (below). A section you wrote by hand works the same way: any [plugins] section gets the drop-in, and removing the section removes the drop-in on the next run.

Installing plugins from the release

--install-plugins NAMES (adf2ae9) installs bcx plugins that a bcd release ships, today only bcx-ipmi, the IPMI BMC driver a site agent hosts (see IPMI through bcx-ipmi). It implies --plugins and needs --release-manifest: the CLI takes each plugin's archive digests, bcx-ipmi_<version>_linux_{amd64,arm64}.tar.gz, from the verified, signed manifest (RELEASE_INCOMPLETE when it lists neither), and refuses any other name.

better-compute daemon install-script --pool pool_site --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.9.1 --release-manifest dist/manifest.json --release-pins sha256/... \
  --install-plugins bcx-ipmi > install-bcd.sh

For each plugin the script:

1. downloads {base-url}{base-path}/downloads/<name>/<version>/linux-<arch> over https (the same version as bcd) and checks the archive against its pinned SHA-256;
2. refuses an archive that names a path outside it (a leading / or a ..), that has no member named after the plugin, or whose member is not a regular file (a symlink or hard link), and extracts only that member;
3. installs it with mode 0755 as /usr/local/lib/bcd/plugins/<name> (written as <name>.new and renamed);
4. writes its binary's SHA-256 as its line in the directory's pins file (0644), replacing an older line for the same plugin and keeping every other line.

A rerun replaces the plugin and its pin without duplicating the line. Like --plugins, it is refused with --unprivileged. The script itself sets up a node (bcd node); a site agent that reads the same config.toml finds the plugin in its [plugins] dir (see a site agent). tests/cli-pki.test.ts runs the install function in a real sh against real archives, including a wrong digest, a symlinked member and a ../ member.

Running bcd node unprivileged

bcd helper (28dc2ae, PLAN.md §6, v0.4) lets bcd node run as an unprivileged account while a root process beside it does the privileged work: the CDI specs, allocation env files and sshd files bcd owns, the bcd_instances and bcd_taps nftables tables, GPU resets, reboots and sshd reloads. Privilege separation describes the operations and the checks. The mode is opt-in: generate the script with --unprivileged (169c459).

better-compute daemon install-script --pool pool_h100 --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.1.0 --sha256 amd64=<hex> --unprivileged > install-bcd.sh

The flag adds a [helper] section to config.toml unless the file already has one. Both processes read that file:

[helper]
socket = "/run/bcd-helper/helper.sock"
user = "bcd"

The socket is not in /run/bcd: the bcd user owns that directory, and a directory the node can write would let it redirect root's work on the socket path with a symlink. So it lives in the helper's own root-owned runtime directory. Since 2d67b25 the helper refuses to start unless the socket's directory is root's and closed to group and other writes, and it binds, chmods and chowns the socket in a private 0700 directory before renaming it into place. Since c139b9c, a helper running as root also refuses to start unless every existing directory from / down to the slots directory, the socket's directory and (since 9e71b1f) the entrypoint's directory is root's and closed to group and other writes. It checks this before it touches its update records or arms a watchdog. A host whose /usr/local/lib is group-writable (root:staff 2775 on older Debian) fails it; bcd doctor, run as root on a helper node, reports each such directory as a failed helper check.

The script decides the mode from the configuration, not the flag: it sets up helper mode exactly when config.toml has a [helper] section, so an upgrade run keeps the node's mode, and deleting the section returns the node to root on the next run (the drop-in is removed; the account and bcd-helper.service are left, and the helper cannot start without the section, so disable that unit yourself). In helper mode it:

1. creates the bcd system account if there is none (useradd --system --no-create-home --home-dir /var/lib/bcd --shell /usr/sbin/nologin bcd, or busybox adduser -S -D -H without useradd);
2. installs bcd-helper.service and the bcd.service drop-in unprivileged.conf, both 0644;
3. creates /usr/local/lib/bcd, where the helper keeps the self-update slots, with mode 0755, and sets an existing one to 0755 (27c9551). The helper's UMask=0077 would otherwise leave it 0700, and bcd node, running as bcd, could not execute the slot its entrypoint links to after the first update. The helper itself creates every missing directory on the way to the slots with mode 0755 whatever its umask, and never changes the mode of one that exists (c139b9c, 9e71b1f);
4. makes /etc/bcd (0750) and config.toml (0640) group-owned by bcd, so the node can read its configuration and nothing else of /etc/bcd becomes readable to it. Both stay root's: since 3fa9021 bcd helper refuses to start as root unless its config.toml is a regular file root owns that no one else can write, in directories only root can write, because that file ([host] units, command_root_sha256, the join pin) decides what root does for the node. Do not hand the file to bcd or make it group-writable;
5. on a host with /etc/libvirt/qemu.conf (d418820): sets dynamic_ownership = 0 there (replacing the line, commented or not, or appending it), adds libvirt-qemu or qemu, whichever exists, to the bcd group, and restarts virtqemud.service or libvirtd.service if it runs. libvirt then chowns no path the node names, and QEMU reaches the node's disk files through the bcd group (see libvirt through the helper);
6. keeps the bcd user away from the TPM (8870b8d, security review 3). The kernel's resource manager filters no command, so any member of the tss group could change the hierarchies' authorization, clear the TPM or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys) and attestation. bcd node therefore never opens /dev/tpmrm0: bcd helper derives, signs with and certifies its node keys (see node keys through the helper). Scripts from 0ba2a79 until then added bcd to tss and wrote a udev rule; an upgrade run takes bcd out of tss when it is a member (gpasswd -d bcd tss, or busybox delgroup bcd tss) and deletes /etc/udev/rules.d/60-bcd-tpm.rules. It does not change the device's current group or mode back. Until that run, a host installed between 0ba2a79 and 8870b8d keeps bcd in tss, and since e6c3f53 bcd doctor fails its tpm check there: "bcd is in the tss group: it can send the TPM any command (lock or clear its hierarchies); bcd helper holds the node keys, so remove it (gpasswd -d bcd tss) and restart bcd.service". Rerun the script, or do that by hand;
7. after bcd join (which runs as root), gives /var/lib/bcd to bcd:bcd and starts bcd-helper.service before bcd.service; an upgrade run does the same with restarts.

[Unit]
Description=better-compute node agent privileged helper (bcd helper)
Documentation=https://github.com/better-compute/better-compute
Before=bcd.service

[Service]
Type=simple
ExecStart=/usr/local/bin/bcd helper --config /etc/bcd/config.toml
Restart=always
RestartSec=2s
UMask=0077
RuntimeDirectory=bcd-helper
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

[Unit]
Requires=bcd-helper.service
After=bcd-helper.service

[Service]
User=bcd
Group=bcd
SupplementaryGroups=systemd-journal adm

The helper serves a caller only when SO_PEERCRED shows bcd's uid and gid (or root) and /proc/<pid>/exe is an installed bcd: [update] entrypoint, bcd-a or bcd-b in [update] slots_dir, or the helper's own binary. A node run from anywhere else is refused.

Self-update works in this mode (2d67b25): the node downloads a release into <data_dir>/staging, and the helper verifies it under its own [update] pins and floor, copies it into root's [update] slots_dir (default /usr/local/lib/bcd) and swaps; see self-update through the helper. Nodes installed by a script generated with --release-pins get the same [update] section as a root node.

What the helper does not do runs with the bcd account's own rights: the journal and the node key file (on a host with a TPM only the key's record; the key itself is used through the helper's tpm.* operations), the status socket, the libvirt-kvm runtime's reads (virsh -r, libvirt's read-only socket) and its files under data_dir/libvirt, the cloud-hypervisor runtime's files, which it shares with each VMM's user through ACL entries, and reading /dev/kmsg. Defining and running libvirt domains goes through the helper's checked libvirt.* operations (ed3c7d2; see libvirt through the helper), and cloud-hypervisor's taps, VFIO groups and VMM units through its ch.* operations (7fe74a1; see cloud-hypervisor through the helper). Only bcd node uses the helper; bcd site ignores [helper]. tests/e2e/bcd.test.ts runs both processes by hand on a fakehw build, a GPU hold and a signed self-update included; tests/cli-pki.test.ts checks the script's section, account, units and drop-in.

Joining

The join itself is bcd join, which the script runs for you:

1. bcd generates a P-256 key and a CSR. On a host with a TPM the key is made in the TPM (50e18f9) and node.key (0600) under data_dir keeps only its record; without one, or when the TPM cannot hold the key, node.key is a PKCS#8 file. See the node key in the TPM.
2. It seals the token to the JoinService encryption key and calls Join on the gateway over TLS pinned to the serving CA, with its hardware claims and the attestor join-token. The gateway relays the call; it can neither read the token nor forge the answer. A deferral is retried with the same CSR; retryable errors back off.
3. It verifies the signed JoinResponse under the pinned signing key, checks that the response is for its own CSR, that the leaf certifies its key, the SPIFFE id and the role, and stores the certificate chain and trust bundle in identity.json. A journal left by an earlier identity is moved aside.

bcd never takes the token from a command-line argument, so a token pasted in the wrong place does not reach shell history or logs. bcd join rejects positional arguments without echoing them:

$ bcd join bcjt_jt_01...
bcd join: takes no positional arguments
bcd join reads the join token from BCD_JOIN_TOKEN or stdin, never from arguments

--token-file PATH reads the token from a file instead (c29bb16). The file must be its owner's alone: mode 0600 or 0400 (a Kubernetes Secret volume with defaultMode: 0400, whose ..data symlink bcd follows), a regular file, and at most 4 KiB; anything else is refused before the token is read. Once the join succeeds, or when the machine had already joined, bcd deletes the file, since a token is single use; on a read-only mount such as a Secret volume it cannot, and logs a warning instead of failing. bcd on Kubernetes joins this way.

The node then opens its session (see sessions) and appears as adopting. Tokens, attestors, refusals and certificates are covered in joining nodes.

Joining again with --reset

bcd join on a machine whose identity loads answers already joined as <node> and exits 0, whatever state the certificate is in: expired, revoked, refused at renewal, or naming an AK this TPM no longer has. That keeps a join unit safe to run on every boot, but it cannot bring such a node back. Since a1c1a46 --reset can:

# as root: see what it would move (moves nothing, exits 2)
bcd join --config /etc/bcd/config.toml --reset
# move the identity aside and join again, here with the TPM
bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm
# or with a new join token on standard input
bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt

What it does and does not do:

• Root only. Outside development (fakehw) builds a non-root --reset fails with "run it as root", exit 1.
• Nothing is deleted. The old files stay in data_dir under their new names, so an operator can still read them (a TPM key's record still names its key). Remove them yourself once the node is back.
• Then the usual join. The move happens after the configuration loads and before the join reads any credential, so the same run needs the join's credential as any join: a token (BCD_JOIN_TOKEN, standard input or --token-file), --attestor tpm, or --node-id for an installed OS. If the join then fails, the identity stays moved aside and a later plain bcd join joins. bcd's own hints spell it as two runs, bcd join --reset --yes, then bcd join. With nothing to move, --reset does nothing and the join goes on.
• Same node or a new one. An attestor that proves the machine (tpm with the same EK, provisioned) reclaims the node it is bound to. A token join of a machine whose trusted hardware key the old node still holds needs a token with allowReclaim (see placeholder hardware and reclaims); to join as a new node instead, cordon and delete the old one first (better-compute nodes delete <node> --yes; see removing an adopted host).
• The journal moves too (b4fcc2e). journal.db is the old node's, and bcd node refuses a journal of another node ("the journal belongs to node <old>, not <new>"). A plain join moves an old journal aside only when the old identity still loads, which after --reset it does not, so --reset moves it with the identity and the node starts a new one.
• Under [helper] the join runs as root and writes the new node.key and identity.json as root. When it finishes, it hands both to the [helper] user with Lchown (b4fcc2e; root only, and a name swapped for a symlink changes only the link), for a new join and for already joined alike, so bcd node can read its key. Then restart bcd-helper.service and bcd.service; re-running the install script also does the restarts.

bcd doctor points at it: the identity check fails for an identity that does not load and for an expired certificate, and both details end with "bcd join --reset --yes, then bcd join". TestJoinReset covers the dry run, the move (the journal included) and that no identity loads afterwards; TestJoinHandsTheIdentityToTheNode covers the hand-over (as root; skipped otherwise).

Site agents and the ramdisk

The install script sets up nodes only: it joins with role node and its unit runs bcd node. The other two agent modes use the same binary but are set up differently.

A site agent

A site agent reads the same config.toml and join-encryption.jwk, joins with a site token and runs bcd site:

BCD_JOIN_TOKEN=bcjt_... bcd join --role site --config /etc/bcd/config.toml
bcd site --config /etc/bcd/config.toml

bcd site refuses a data directory whose identity has another role. Without a [boot] section it runs BMC operations, site.httpProxy and boot entries but serves no PXE machine (it logs a warning). With one, it also runs the boot server:

gateway_url = "https://gw.eu.example.com"
site = "site_01k5w..."
serving_ca_spki_pin = "sha256/..."
join_signing_key_pin = "sha256/..."
join_encryption_key_pin = "sha256/..."

[boot]
listen = "0.0.0.0:8080"
url = "http://10.20.0.2:8080"
kernel = "http://10.20.0.2:8080/ramdisk/vmlinuz"
initrd = "http://10.20.0.2:8080/ramdisk/initrd"
cmdline = "console=ttyS0,115200"
server_ip = "10.20.0.2"
proxy_dhcp = true
tftp = ":69"
tftp_root = "/srv/bcd/ipxe"

proxy_dhcp and tftp need server_ip and url, or bcd site exits 2. The iPXE builds and the ramdisk kernel and initramfs are yours to supply: the repository builds neither. Ports 67, 69 and 4011 are privileged, and the unit the install script writes keeps bcd to a capability set without CAP_NET_BIND_SERVICE, so a site agent needs its own unit. What the boot server answers, and when, is on sites.

The ramdisk

bcd ramdisk is not installed. It runs inside the PXE ramdisk the site agent boots, and takes everything from the kernel command line the site agent rendered into the machine's iPXE script, so it has no config.toml:

A missing parameter exits 2. The ramdisk joins with the discovery attestor as role ramdisk (see attestation), logs JSON, then runs the agent until the machine kexecs into its installed OS or powers off. Its image needs bcd plus the tools its kinds call: nvme-cli, hdparm, util-linux (blkdiscard, blockdev, lsblk, mount), e2fsprogs' debugfs and mtools (to write into and read from a deployed image's ext and FAT filesystems without mounting them; 1ae1789, d9a900f), udevadm, kexec-tools and, for diag.dcgm, dcgmi.

Upgrades

Before v0.4: rerun the script

Run the same script, or one generated for a newer --bcd-version, without a token on a host that already has /etc/bcd/config.toml. It downloads and verifies the new binary, replaces /usr/local/bin/bcd with an atomic rename, rewrites the key, the unit and the top-level keys of the configuration, and restarts the service:

sudo sh install-bcd.sh
# bcd upgraded to 0.1.1 and restarted.

The configuration's top-level keys (gateway, site and the three pins) come from the new script, and every section of the existing file ([plugins], [boot], [imds], [journal], [log], ...) is kept as it was (4225772; earlier scripts rewrote the file whole and dropped them). Top-level keys you added by hand, such as data_dir or gateway_server_name, are not kept, so put them back after an upgrade or regenerate the script. The plugin drop-in follows the kept configuration: a node with [plugins] keeps it, even when the new script was generated without --plugins.

The node keeps its identity and journal across the restart. On reconnect its Hello carries the new daemon_version, which the gateway records in node_sessions.daemon_version.

On a node that has already self-updated, the script's mv -f replaces the entrypoint symlink with a plain file. The next self-update adopts that file into the current slot before it relinks the entrypoint (9006eee), so it never falls back to the older binary the slot held; before that fix it did, rolling the script's upgrade back.

Upgrading an unprivileged libvirt host

A VPS or sandbox host installed with --unprivileged by a script from before d418820 runs bcd node in the libvirt group, which is root-equivalent, and its libvirt still relabels files. A new bcd there sends domain operations through the helper anyway (ed3c7d2), which refuses to define a domain with a NIC until [helper] instance_bridges is set, but the group membership and the relabelling stay until you run a newer script, generated with --unprivileged (and --instance-bridges for the instances' bridges), on the host without a token. It rewrites the unprivileged.conf drop-in without SupplementaryGroups=libvirt, sets dynamic_ownership = 0, adds QEMU's user to the bcd group, restarts libvirt and then both bcd units:

better-compute daemon install-script --pool pool_vps --site fsn1 \
  --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \
  --bcd-version 0.9.2 --sha256 amd64=<hex> --unprivileged --instance-bridges br0 > install-bcd.sh
sudo sh install-bcd.sh

Without [helper] instance_bridges the helper defines no domain with a NIC, so set the bridges before instances with networks start there; the script adds them only when the configuration has no instance_bridges line yet. bcd doctor's libvirt check fails until the host is set up this way.

From v0.4: rollouts and self-update

The v0.4 design moves upgrades into desired state. A rollout sets NodeSpec.daemon = { version, url, sha256, signature } on nodes wave by wave (1%, 10%, 50%, 100% by default) within each pool's disruption budget, and a node succeeds when it rejoins reporting that version. A daemon too old to speak the server's minimum protocol version gets Welcome.upgrade_required, and its session accepts only the daemon.upgrade command kind, which carries the same DaemonSection; the rollout controller sends it to such nodes.

Self-update

bcd updates itself in place (de9f380, daemon/internal/agent/update.go, daemon/internal/selfupdate), for both the daemon section and daemon.upgrade (node and site roles). Since 2d67b25 the privileged half (selfupdate.Manager: verify, swap, confirm, roll back) runs in process on a root node and inside bcd helper on an unprivileged one; see self-update through the helper. The steps are the same:

1. Only a signed release. DaemonSection.signature is the release's signed manifest (manifest.json from better-compute release sign). At least [update] release_threshold distinct keys pinned in [update] release_pins must have signed it (purpose release-manifest), and it must be for product bcd and the section's version, have a sequence not below the release this node ran or was installed at ([update] release_sequence; no rollback), be unexpired on both the host clock and server time, and list the bare binary bcd_<version>_<os>_<arch> with the section's SHA-256. Without release pins bcd takes no binary at all: the control plane alone can never hand a node one. The URL must be https://.
2. Two slots. The binaries live in bcd-a and bcd-b of the slots directory ([update] slots_dir: default <data_dir>/bin, or /usr/local/lib/bcd under the helper), with current and previous symlinks and slots.json. The first update adopts the binary the install script put at the entrypoint ([update] entrypoint, default /usr/local/bin/bcd) into slot a and makes the entrypoint a symlink to current; an entrypoint that is a symlink somewhere else is refused, and a plain file found there later (a rerun of the install script) is copied into the current slot first (9006eee). Running as root, adoption takes only a root-owned regular file that group and others cannot write (9e71b1f). The node downloads the release into <data_dir>/staging/bcd-<version>, at most 512 MiB, checked against the manifest's size and digest as it streams, with an idle watchdog; the swap then copies it into the other slot, checking the size and digest again as it writes. It opens the staged file beneath the staging directory's parent (os.Root), refuses a symlinked staging directory, opens with O_NOFOLLOW|O_NONBLOCK (a FIFO cannot hang it), and takes only a regular file with one link; under the helper the file must also be owned by the node's uid (c139b9c).
3. Swap and confirm. bcd writes upgrade.json with a deadline of now plus 120 seconds (fsynced), points current at the new slot and execs the entrypoint once everything is closed. The new binary confirms itself on its first Welcome: the release floor moves up (update.json) and upgrade.json goes. Without a Welcome by the deadline it swaps back and restarts; the old binary then records the version as failed ("bcd X did not start"), reports the daemon section FAILED with reason UpdateFailed, and never tries that version again until bcd update --clear-failed. Under the helper, the node asks the helper to confirm, and the helper's own watchdog also swaps back an update still unconfirmed 30 seconds after its deadline and restarts bcd.service.
4. Crash loops. Under systemd the first update writes bcd-rollback.service and a bcd.service drop-in (50-bcd-rollback.conf: StartLimitIntervalSec=600, StartLimitBurst=5, OnFailure=bcd-rollback.service), which restores previous, only while upgrade.json exists. fakehw builds never write them.
5. No going back below the floor. A confirmed update is never undone (c139b9c). slots.json records each slot's version, manifest sequence and the SHA-256 of the bytes written into it. The binary already in the other slot is taken without a manifest only when its recorded sequence is known and not below the release floor, so a section that names the older version the node ran before is refused: without a manifest there is nothing to verify, and that release's manifest is below the floor (the daemon section reports FAILED). An update that never confirmed is still swapped back by the deadline, the crash-loop unit or the helper's watchdog. To undo a release that confirmed, roll forward: sign a newer release with a higher sequence and roll it out (rollouts).

daemon.upgrade is self-terminating, like node.reboot: it reports RUNNING before the swap and SUCCEEDED from the new binary once it is welcomed, or FAILED with update-failed after a swap back; a restart without a reboot is decided by upgrade.json, not the boot id. Running workloads are not touched. bcd sim and bcd ramdisk run without self-update: the daemon section stays SectionUnsupported and the kind UNSUPPORTED. bcd update shows the slots, the release floor, a pending update and failed versions from [update] slots_dir (run it as root on a helper node, whose records are root's), and bcd doctor warns without release pins, fails when the slots are in use but the entrypoint does not link to current, and warns about a pending update or a failed version (2845f41). Both find the slots and records the same way (8eca2b1): [update] slots_dir, else /usr/local/lib/bcd under the helper, else <data_dir>/bin with the records in data_dir. tests/e2e/bcd.test.ts updates a real bcd to a second signed build through daemon.upgrade, a real exec and its Welcome (fc0ca6a), and delivers a signed build to an unprivileged node through bcd helper (2d67b25); no release has been cut, so no real host has self-updated yet. See delivery.

The fleet-wide order for any upgrade is: expand migrations, then workers and api, then gateways one at a time with Drain, then daemons in waves, and contract migrations in the next release. See version skew.

Removing bcd

Neither the CLI nor bcd has an uninstall step, and PLAN.md designs none. Retire the node in the control plane first so nothing is scheduled on it (see operating nodes). Then stop and disable bcd.service and remove the files listed above.

Release artifacts

bcd is configured to build with goreleaser from daemon/ (.goreleaser.yaml) for Linux on amd64 and arm64, with cgo and zig cc, set up for a go-nvml backend that would load libnvidia-ml; the GPU code in the tree today reads nvidia-smi and needs no cgo (cb987e5). A release produces bcd_<version>_linux_{amd64,arm64}.tar.gz, each with a .sha256, and the bare binaries bcd_<version>_linux_{amd64,arm64}. The bare binary is what the install script downloads, what a signed release manifest lists and what self-update verifies. The same release builds bcx-ipmi, the site agent's IPMI BMC plugin, without cgo and ships it only as bcx-ipmi_<version>_linux_{amd64,arm64}.tar.gz (348ff56), so a release still has exactly one bare binary per platform, bcd's, which rollouts and self-update take. goreleaser also writes an SPDX SBOM per archive (syft) and a keyless cosign signature and certificate for every artifact. .github/workflows/release.yml runs it on a v* tag (or a dispatch naming one), appends everything to that tag's GitHub release, and adds SLSA level 3 provenance (bcd.intoto.jsonl) from the generic SLSA generator. No tag has been released yet; CI (.github/workflows/daemon.yml) vets, tests and runs goreleaser check, and a local snapshot build is goreleaser release --snapshot --clean --skip=sign,sbom.

Until then a binary comes from a plain go build in daemon/. The development and test builds are cross-compiled without cgo (GOOS=linux GOARCH=amd64 CGO_ENABLED=0); node scripts/daemon-test.mjs --build-bcd writes the fake hardware build to daemon/bin/bcd-linux-amd64. go-nvml is not a dependency, so a cgo build gains nothing today: the default build reads GPUs through nvidia-smi without cgo (cb987e5).