# Installing bcd (/docs/daemon/install) > The install script that puts bcd on a server, the files and hardened systemd unit it writes, joining with BCD_JOIN_TOKEN, and upgrades. A server becomes a node in two moves: an operator generates an install script once per pool, then runs it on each machine with a single-use join token. The script downloads a pinned bcd binary, checks its digest, writes a small configuration, installs a hardened systemd unit, joins the fleet and starts the daemon. The script contains no secret, so you can keep it in a provisioning repository; the token travels separately. > **The pieces are not all runnable yet.** The script generator (`better-compute daemon install-script`) is built and tested, and so is the `GET {basePath}/downloads/bcd/{version}/linux-{arch}` route it fetches from (see [serving the binaries](#serving-the-binaries)). bcd itself is built and tested: `bcd join` and `bcd node` work, and `tests/e2e/bcd.test.ts` runs them against a real gateway (see [the overview](/docs/daemon#what-bcd-does-today)). But bcd is not released: the release workflow exists, but no tag has been released, so there are no published bare binaries and digests for the script to pin, and no test yet runs the script itself on a host. You can build a binary from `daemon/` and serve it yourself; otherwise `better-compute dev --sim N` shows nodes joining through the same protocol with the TypeScript FakeDaemon. See [local development](/docs/operations/local-development). ## Serving the binaries [#serving-the-binaries] The api serves the daemon binaries itself when you configure `downloads`: ```ts betterCompute({ // … downloads: { dir: '/srv/bcd' }, // or { blob: blobStore }, or both (the blob store is tried first) }); ``` `GET {basePath}/downloads/bcd/{version}/{os}-{arch}` answers goreleaser's bare binary `bcd_{version}_{os}_{arch}`, read from `downloads.blob` under the key `bcd/{version}/{file}` or from `{dir}/{file}`. The route is public: the install script and self-update check the sha256 from the verified release manifest, not the transport. Responses are `application/octet-stream` with `cache-control: public, max-age=31536000, immutable`. The path must match a strict pattern (a semantic version with an optional leading `v`, `linux`, `amd64` or `arm64`), so nothing else on disk can be reached; anything else, or a binary the deployment does not have, is 404. `downloads` needs `dir` or `blob`, or `betterCompute()` throws `INVALID_CONFIG`. The same route serves the `bcx-ipmi` plugin (97b2ac2): `GET {basePath}/downloads/bcx-ipmi/{version}/{os}-{arch}` answers goreleaser's archive `bcx-ipmi_{version}_{os}_{arch}.tar.gz`, from the blob key `bcx-ipmi/{version}/{file}` or `{dir}/{file}`. The products are a fixed list, `bcd` and `bcx-ipmi`; any other name is 404. The route stays public because the archive's digest is in the same signed release manifest, and the install script checks the download against it ([installing plugins from the release](#installing-plugins-from-the-release)). > **Where the binaries come from.** The route only serves files you put there. Copy each release's bare binaries (from goreleaser's output) into the directory or blob store before you roll the version out. ## Generate the script [#generate-the-script] `daemon install-script` reads the public half of the PKI bundle for the pins, so it needs no secret. Create the bundle first with `better-compute pki init` (see [PKI](/docs/operations/pki)). ```bash better-compute daemon install-script \ --pool pool_h100 --site fsn1 \ --gateway-url https://gw.eu.example.com \ --base-url https://compute.example.com \ --bcd-version 0.1.0 \ --sha256 amd64=,arm64= > install-bcd.sh ``` | Flag | Required | Environment | Meaning | | ------------------------------- | --------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--pool ID` | yes | | The pool the node joins; named in the script's header | | `--site ID` | yes | | The pool's site; written to `config.toml` as `site` | | `--gateway-url URL` | yes | `BETTER_COMPUTE_GATEWAY_URL` | The regional gateway bcd dials, `https://...` | | `--base-url URL` | yes | `BETTER_COMPUTE_BASE_URL` | The control plane that serves the downloads | | `--base-path PATH` | no | | The API base path, default `/api/compute` | | `--bcd-version VERSION` | yes | `BETTER_COMPUTE_BCD_VERSION` | The release to install, e.g. `0.1.0` (a leading `v` is dropped) | | `--sha256 ARCH=HEX,...` | one of two | | SHA-256 of each architecture's bare binary | | `--release-manifest PATH` | one of two | | A signed release manifest; its verified digests are pinned instead | | `--release-pins PINS` | with a manifest | `BCD_RELEASE_PINS` | Comma-separated `sha256/...` pins of the release keys; also written to `config.toml` as `[update] release_pins`, which [self-update](#self-update) needs | | `--release-threshold N` | no | | How many release keys must have signed (1 to 16, default 1); `[update] release_threshold` | | `--release-sequence N` | no | | The node's release floor, `[update] release_sequence`: bcd never installs an older release; needs `--release-pins` | | `--websocket-url URL` | no | | For networks whose TLS-inspecting proxies break mTLS: a `wss://.../v1/session` URL, written as the `[websocket]` section so bcd runs its session and renewals over the [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04) (0ea4ab4). A configuration that already has a `[websocket]` section is left alone, and the script says so | | `--websocket-audience HOST` | no | | `[websocket] audience`, the gateway's public host the node's proof names; needs `--websocket-url` | | `--proxy-ca-file PATH` | no | | The TLS-inspecting proxy's CA, read from the local file and embedded only when it holds PEM certificates and nothing else; the script writes it to `/etc/bcd/websocket-proxy-ca.pem` and sets `[websocket] proxy_ca_file`. It is trusted for the WebSocket's TLS only: the daemon still proves its identity inside the session, and the gateway's proof must chain to the pinned serving CA. Needs `--websocket-url` | | `--bmc-networks CIDRS` | no | | A site agent's BMC networks (comma-separated), written as `[site] bmc_networks`: `site.httpProxy` never reaches into them | | `--plugins` | no | | Prepare the node for bcx driver plugins: a `[plugins]` section and the systemd drop-in; see [the plugin drop-in](#the-plugin-drop-in) | | `--privileged-plugins NAMES` | no | | Comma-separated plugins that run as root without the seccomp filter, such as a Kata runtime; implies `--plugins` | | `--install-plugins NAMES` | no | | Comma-separated bcx plugins to install from the release, each checked against its archive digest in the verified `--release-manifest` (adf2ae9). Only plugins a release ships are accepted: `bcx-ipmi`. Implies `--plugins` and needs `--release-manifest`; see [installing plugins from the release](#installing-plugins-from-the-release) | | `--unprivileged` | no | | Run `bcd node` as the unprivileged `bcd` user beside a root `bcd helper` (169c459): a `[helper]` section, the `bcd` account, `bcd-helper.service` and a `bcd.service` drop-in; see [running bcd node unprivileged](#running-bcd-node-unprivileged). Refused with `--plugins`, `--privileged-plugins` or `--install-plugins`. Self-update works in this mode: the helper does the privileged half (2d67b25). On a `libvirt-kvm` host the helper defines and runs the domains, and the `bcd` account stays out of the `libvirt` group (d418820) | | `--attestor tpm` | no | | Join with the host's TPM instead of a join token (a006d9f): the script reads no token, fails early without `/dev/tpmrm0`, and runs `bcd join --attestor tpm`; see [joining with the TPM](#joining-with-the-tpm). `tpm` is the only value; any other is `INVALID_INPUT` | | `--instance-bridges BRIDGES` | no | | With `--unprivileged`, on VPS and sandbox hosts: the bridges an instance's NIC may join, comma-separated interface names (1 to 15 of letters, digits, `.`, `_` and `-`), written as `[helper] instance_bridges` unless the configuration already sets it (d418820). Without `--unprivileged` it is `INVALID_INPUT` | | `--approver-set-sha256 HEX` | no | | Writes `approver_set_sha256` (64 lower-case hex digits, the `approverSetSha256` that `pki trust-version` prints) into `config.toml`, pinning the node's first approver set (9b7a199): a keyless bcd takes only that set, and runs nothing destructive until it has it; see [pinning the first set](/docs/operations/pki#pinning-the-first-set). An upgrade run without the flag keeps the node's existing pin (b6d401f) | | `--command-root-sha256 HEX,...` | no | | Writes `command_root_sha256`, 1 to 16 lower-case hex SHA-256 digests of the command-signing root certificates (`pki trust-version` prints them as `commandRootSha256`), so `bcd helper` checks host config units against specs signed under them from its first request (59985ae; see [signed specs](/docs/daemon/host-config#signed-specs)). Without the flag the script pins the PKI bundle's command-signing root and, during a rotation, the added one, and nothing when the bundle has no command signing; an upgrade by a script without pins keeps the node's | | `--pki PATH` | no | | The PKI bundle to read the pins from | | `--out PATH` | no | | Write a new file with mode 0755 instead of printing; refuses to overwrite (`FILE_EXISTS`) | Give exactly one of `--sha256` and `--release-manifest`. With a manifest, the CLI verifies its signatures under the pins and threshold, checks that it is for product `bcd` and for `--bcd-version` (`RELEASE_MISMATCH` otherwise), and pins the one bare `linux-amd64` and `linux-arm64` binary it lists; archives, checksums and metadata files are skipped. See [rollouts](/docs/guides/rollouts) for signing releases. Every value embedded in the script is checked against a narrow pattern before it is written, so nothing a caller passes can break out of the shell or TOML quoting: | Value | Must be | | ------------------------------------------ | ------------------------------------------------------------------------------------------ | | pool and site ids, privileged plugin names | letters, digits, `.`, `_` and `-`, starting with a letter or digit, at most 128 characters | | version | a semantic version such as `0.1.0` or `v0.2.0-rc.1` | | pins | `sha256/` followed by the standard base64 of 32 bytes | | digests | 64 lower-case hex digits, for `amd64` and/or `arm64` only | | URLs | `https://` with letters, digits and `. - _ ~ /` only; trailing slashes are dropped | | join encryption key | a P-256 public JWK (`kty: EC`, `crv: P-256`) | A bad value fails with `INVALID_INPUT` naming the flag. ## What the script does [#what-the-script-does] The output is a POSIX `sh` script that runs with `set -eu` and `umask 077`. ### Check the host [#check-the-host] It stops unless the host runs Linux, the caller is root, and `systemctl` and `sha256sum` exist. It maps `uname -m` to an architecture (`x86_64` or `amd64` to `amd64`, `aarch64` or `arm64` to `arm64`) and refuses an architecture the script carries no digest for. ### Read the token [#read-the-token] The token comes from `BCD_JOIN_TOKEN`, or else from the first line of standard input when standard input is not a terminal. The variable is unset right after. Anything that does not look like `bcjt_.` is refused with exit code 2. Without a token, the script continues only when `/etc/bcd/config.toml` already exists; that is an upgrade (below). ### Download and verify [#download-and-verify] It fetches `{downloads}/bcd/{version}/linux-{arch}` into a temporary directory with `curl` (HTTPS only, TLS 1.2 or later, three retries) or `wget --https-only`, and compares the file with the pinned SHA-256. A mismatch stops the script before anything is installed. ### Install the files [#install-the-files] It creates `/etc/bcd` and `/var/lib/bcd` with mode 0700 and `/etc/cdi` with mode 0755 (and, with `--plugins`, `/usr/local/lib/bcd/plugins` with mode 0755). Each file is written next to its target as `*.new` and renamed over it, so a replacement is atomic even while bcd runs. When a `config.toml` already exists, every section of it (`[plugins]`, `[boot]`, `[imds]`, ...) is carried over below the freshly written top-level keys; with `--plugins`, a `[plugins]` section is appended unless the file already has one, and with `--unprivileged` a `[helper]` section likewise. After the unit, it installs or removes the [plugin drop-in](#the-plugin-drop-in) according to whether `config.toml` has a `[plugins]` section, and the [helper mode](#running-bcd-node-unprivileged) (the `bcd` account, `bcd-helper.service`, the `unprivileged.conf` drop-in, `/etc/bcd` readable by the `bcd` group) according to whether it has a `[helper]` section. Then `systemctl daemon-reload`. ### Check, join and start [#check-join-and-start] It runs `bcd doctor --config /etc/bcd/config.toml` and prints a warning if doctor reports a problem, without stopping. Before the first join, doctor's `identity` check only warns that the node has not joined; a gateway it cannot reach over TCP is a failure. With a token, it pipes the token into `bcd join --config /etc/bcd/config.toml` with `printf`, a shell builtin, so the token never appears in a process listing, then runs `systemctl enable --now bcd.service`. Without a token, it runs `systemctl restart bcd.service`. In helper mode it first hands `/var/lib/bcd` to the `bcd` user (`chown -R bcd:bcd`, since `bcd join` wrote the identity as root) and enables and starts, or restarts, `bcd-helper.service`, so the helper is up before the node. Download the script before running it. Piping it into `sh` would leave no standard input for the token. ```bash BCD_JOIN_TOKEN=bcjt_... sudo -E sh install-bcd.sh # or, keeping the token out of the environment: sudo sh install-bcd.sh < token-file ``` ### Joining with the TPM [#joining-with-the-tpm] A script generated with `--attestor tpm` (a006d9f) joins the machine with its TPM, through the control plane's [`tpm` attestor](/docs/guides/nodes/attestation#tpm), and carries no token step. Its header says the attestor's policy decides where the machine joins. It differs in two places: * **Instead of reading a token**, it unsets `BCD_JOIN_TOKEN` and stops unless `/dev/tpmrm0`, the kernel's TPM resource manager, exists. * **Instead of the token join**, it runs `bcd join --config /etc/bcd/config.toml --attestor tpm < /dev/null`, which returns at once on a machine that already joined, so the same script installs and upgrades. Then it enables and restarts `bcd-helper.service` (in helper mode, after `chown -R bcd:bcd /var/lib/bcd`) and `bcd.service`. As root on a node with `[helper]`, the join also keeps the signed join record the helper binds host units to (see [signed specs](/docs/daemon/host-config#signed-specs)). The join makes the node key in the same TPM and sends the TPM's certification of it, which `tpmAttestor` requires by default (`nodeKey: 'certified'`, cf24876). The join runs as root and opens the TPM itself, with or without `[helper]`; under `[helper]` the unprivileged `bcd node` then reaches that key only through `bcd helper` (8870b8d), and never opens `/dev/tpmrm0` ([below](#running-bcd-node-unprivileged)). If the TPM cannot make the key, the join stops with an error. See [bcd with its TPM](/docs/guides/nodes/attestation#bcd-with-its-tpm). ```bash better-compute daemon install-script --pool pool_tpm --site fsn1 \ --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \ --bcd-version 0.1.0 --sha256 amd64= --attestor tpm > install-bcd.sh sudo sh install-bcd.sh ``` The control plane must run `tpmAttestor` with a `resolve` allow-list or `enrollment` for the machine's endorsement key, else the join is refused or waits for approval; `--pool` only names the pool in the header. A pool can then refuse token joins with `policy.joinAttestors: ['tpm']` (see [requiring an attestor](/docs/guides/nodes/attestation#requiring-an-attestor)). ## Files on the host [#files-on-the-host] - `bcd (0755)` - `config.toml (0600)` - `join-encryption.jwk (0644, public)` - `bcd.service (0644)` - `bcd-helper.service (0644, only on nodes with [helper])` - `plugins.conf (0644, only on nodes with [plugins])` - `unprivileged.conf (0644, only on nodes with [helper])` - `pins (sha256sum format) and the plugin binaries, yours to add` - `node.key (0600, the node key, or its TPM record)` - `identity.json (certificate chain, trust bundle, server time mark)` - `journal.db (the bbolt journal)` - `host-intent.json (0600, the signed spec that last set the host section)` - `volumes/ID.raw, ID.json (0600, persistent host volumes)` - `bootargs/UNIT, hugepages-1g-boot, cdi-nvidia.stamp (the host units' records)` - `k8s-join.json, k8s/bootstrap-token (k8s-join)` - `helper-intent.json (bcd helper's accepted signed spec)` - `libvirt/ (bcd helper's staging for domain definitions and memory saves)` - `bcd.sock (0600 status socket, owned by the user bcd runs as)` - `helper.sock (0660, root:bcd)` - `bcd-ALLOCATION.json (written by the devices converger)` The script writes the binary, `config.toml`, `join-encryption.jwk`, the unit and, on a plugin node, the drop-in and the (empty) plugin directory. The other paths belong to the daemon: `bcd join` writes the key and `identity.json` under `data_dir`, `bcd node` opens the journal there, serves the status socket at `socket_path`, and writes CDI specs for `hold` bindings (with env files under `/run/bcd/allocations`). On a host with a TPM (`/dev/tpmrm0`) the key is made in the TPM and `node.key` holds only its record, a PEM block `BCD TPM NODE KEY`; otherwise it is a PKCS#8 file (see [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm)). A renewal writes the next key as `node.key.next` and promotes it only once the new certificate is stored. A journal that does not open is moved aside as `journal.db.corrupt-` and a new one starts. The token is consumed by the join and never written anywhere. With `--proxy-ca-file` the script also writes `/etc/bcd/websocket-proxy-ca.pem`. After its first [self-update](#self-update) bcd owns `/bin/bcd-a`, `bcd-b`, `current`, `previous` and `slots.json`, `upgrade.json` while an update awaits its `Welcome`, `update.json` (the release floor and failed versions), and, under systemd, `bcd-rollback.service` and `bcd.service.d/50-bcd-rollback.conf`; `/usr/local/bin/bcd` then links to `current`. Under `bcd helper` the slots and both records are in root's `[update] slots_dir` instead (default `/usr/local/lib/bcd`), and the node downloads into `/staging`. A quorum node keeps `batch-leaves.json` and `unbatched-destructive.json`, and a site agent `bmc-hosts.json`, under `data_dir`. ### config.toml [#configtoml] ```toml title="/etc/bcd/config.toml" # Written by the better-compute install script. bcd refuses unknown keys. gateway_url = "https://gw.eu.example.com" site = "fsn1" serving_ca_spki_pin = "sha256/..." join_signing_key_pin = "sha256/..." join_encryption_key_pin = "sha256/..." ``` bcd reads the file with `daemon/internal/config`, a strict subset of TOML 1.0: bare keys, basic or literal strings, decimal integers (with `_` between digits) and booleans, in the top level or a bare `[section]`. Arrays, inline tables, floats, dates, multi-line strings, dotted or quoted keys, unknown keys, unknown sections, duplicate keys and wrong types are all errors with a line number, so a typo fails loudly. Files over 64 KiB are refused. | Key | Required | Default | Meaning | | -------------------------------------------- | ---------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `gateway_url` | yes | | An `https://` URL without credentials | | `site` | yes | | The site id, no whitespace | | `serving_ca_spki_pin` | yes | | Pins the gateway serving CA | | `join_signing_key_pin` | yes | | Pins the JoinService key that signs `JoinResponse` | | `join_encryption_key_pin` | yes | | Pins the JoinService key the token is sealed to | | `data_dir` | no | `/var/lib/bcd` | Absolute path for the journal and key | | `socket_path` | no | `/run/bcd/bcd.sock` | Absolute path of the status socket | | `[journal] max_bytes` | no | 67108864 | May lower the 64 MiB cap, never raise it | | `[journal] max_events` | no | 10000 | May lower the 10,000 unacked-event cap, never raise it | | `[log] level` | no | `info` | `debug`, `info`, `warn` or `error` | | `[log] json` | no | `true` | JSON lines | | `gateway_server_name` | no | | The name the gateway certificate must carry when `gateway_url` names an IP address (an L4 balancer, a dev stack reached from WSL) | | `dev_insecure_h2c` | no | `false` | Allows an `http://` `gateway_url` (cleartext HTTP/2 to a dev gateway). Only fakehw builds accept it; the host build refuses to start | | `approver_set_sha256` | no | | 64 lower-case hex digits: the only approver set a daemon without approver keys takes first (see [pinning the first set](/docs/operations/pki#pinning-the-first-set)); `--approver-set-sha256` writes it | | `command_root_sha256` | no | | Comma-separated lower-case hex SHA-256 digests of the command-signing root certificates, at most 16 (03c023d); `pki trust-version` prints them as `commandRootSha256`. `bcd helper` then applies a `host.unit` only with a spec signed under these roots, which decide alone. `daemon install-script` writes it by default from the PKI bundle's current command-signing root and, during a rotation, the next one; `--command-root-sha256` overrides that. Without it the helper takes the roots from the `SignedTrustBundle` the JoinService key signed, checked against `join_signing_key_pin` (2d18327): it refuses a bundle older than one it already accepted (a rollback) and, once it has accepted a bundle with roots, any request without a signed spec; before that it checks nothing. A malformed value stops bcd from starting. See [signed specs](/docs/daemon/host-config#signed-specs) | | `[update] release_pins` | no | none | Comma-separated `sha256/...` pins of the release keys. Without them bcd refuses every self-update; see [self-update](#self-update) | | `[update] release_threshold` | no | `1` | How many distinct pinned keys must have signed a release | | `[update] release_sequence` | no | `0` | The release the node was installed at: no release manifest with a lower sequence is taken (0: unknown) | | `[update] entrypoint` | no | `/usr/local/bin/bcd` | The path `bcd.service` starts; the first update makes it a symlink to `current` in the slots directory. Under the helper it may not be inside `data_dir` ("under bcd helper, update.entrypoint may not be inside data\_dir"; 9e71b1f), since root adopts the binary found there into a slot | | `[update] slots_dir` | no | `/bin`; under `bcd helper` `/usr/local/lib/bcd` | An absolute path for the binary slots and, beside them, `upgrade.json` and `update.json` (without it and without the helper, the records stay in `data_dir`; 2d67b25). Under the helper it may neither be inside `data_dir` nor contain it ("under bcd helper, update.slots\_dir may not be inside data\_dir"; both paths cleaned first, c139b9c), since the unprivileged node owns that directory; the helper creates it | | `[helper] socket` | no | | An absolute path: `bcd node` then runs unprivileged and sends its privileged operations, self-update's included, to `bcd helper` on this unix socket (28dc2ae); `--unprivileged` writes `/run/bcd-helper/helper.sock`. The directory must be root's with no group or other write. See [running bcd node unprivileged](#running-bcd-node-unprivileged) | | `[helper] user` | with `socket` | | The unprivileged account `bcd node` runs as (`bcd` from `--unprivileged`); `root` is refused ("helper.socket needs an absolute path and helper.user, the unprivileged account bcd node runs as") | | `[helper] instance_bridges` | no | | Comma-separated bridges an instance's NIC may join (fea4d2e). `bcd helper` refuses to define a libvirt domain with a NIC on any other bridge, and with none set it defines no domain with a NIC at all. A name that is not an interface name stops bcd from starting ("helper.instance\_bridges: … is not an interface name"). `--instance-bridges` writes it; see [libvirt through the helper](/docs/daemon#libvirt-through-the-helper) | | `[site] bmc_networks` | no | | A site agent's BMC networks, comma-separated CIDRs: `site.httpProxy` never reaches into them, whatever the control plane asks. A site agent with it advertises the capability `site.bmc-networks`, and `better-compute doctor` warns about connected site agents without it (a019c85) | | `[websocket] url` | no | | A `wss://` URL (`ws://` only with `dev_insecure_h2c`), without credentials, of the gateway's WebSocket endpoint, `.../v1/session`: `bcd node` and `bcd site` then run the session and renewals over the [WebSocket fallback](/docs/daemon/protocol#websocket-fallback-v04) | | `[websocket] proxy_ca_file` | no | | An absolute path to a PEM file of a TLS-inspecting proxy's CA, trusted for the TLS connection only, never for the gateway's proof | | `[websocket] audience` | no | `gateway_server_name`, else the URL's host | The gateway's public host the node's proof names | | `[instances] balloon_floor_percent` | no | `50` | 10 to 100: the share of its memory a ballooned guest always keeps (12e9ced); bcd also never takes a guest below its use plus 10%. Outside the range bcd does not start. See [memory ballooning](/docs/daemon/delivery#memory-ballooning) | | `[imds] listen` | no | | Where a VPS host serves its instances' metadata service, for example `169.254.169.254:80` on the instances' bridge; empty turns it off | | `[host] units` | no | `true` | `false` when something else manages the host's configuration (the NVIDIA GPU operator beside [bcd as a DaemonSet](/docs/daemon/kubernetes)): bcd applies no `NodeSpec.host` unit and reports the section `UNSUPPORTED` with reason `HostManagedExternally` (a193249). See [host configuration](/docs/daemon/host-config) | | `[kubernetes] allowed_clusters` | for a `k8s-join` | | Comma-separated `https://host[:port]#sha256:` entries: the only clusters the `k8s-join` unit may join this machine to (7579184). Since 9f860ad it is required as root too: without it every join is refused, and `bcd doctor` fails a joined node without one. A malformed entry stops `bcd node` and `bcd helper` from starting. See [k8s-join](/docs/daemon/host-config#k8s-join) | | `[kubernetes] gpu_api` | no | `device-plugin` | How a node bcd did not join itself (the cluster's DaemonSet) offers bound GPUs to the kubelet: `device-plugin` or `dra`, never both (7d24836); any other value stops bcd from starting. A node `k8s-join` joined takes the unit's `gpuApi` instead. See [GPUs for pods](/docs/daemon/kubernetes#gpus-for-pods) | | `[boot] ...` | no | | A site agent's boot server; see [site agents](#a-site-agent) | | `[plugins] dir` | no | | A directory of bcx driver plugins and their `pins` file (sha256sum format); only pinned plugins run. Empty runs none. See [driver plugins in bcd](/docs/extending/drivers#driver-plugins-in-bcd) | | `[plugins] user` | no | | Who plugins run as when bcd is root (no supplementary groups); unset, each plugin gets its own uid from a block bcd keeps, `0x7FFE0000` to `0x7FFEFFFF`. `root` is refused | | `[plugins] privileged` | no | | Plugins (comma-separated names) that run as root without the shim or the filter, such as a Kata runtime | | `[plugins] sandbox` | no | `true` | Runs unprivileged plugins through bcd's `__bcx-exec` shim: `no_new_privs` and a seccomp filter. Linux only; elsewhere set `false` or no plugin starts | | `[plugins] cgroup` | no | | An absolute path under `/sys/fs/cgroup` in bcd's delegated cgroup; bcd creates it and enables the memory and pids controllers, and each plugin runs in its own child `bcx-`. `--plugins` writes `/sys/fs/cgroup/system.slice/bcd.service/plugins` | | `[plugins] memory_max`, `[plugins] pids_max` | no | 0 (none) | The child cgroup's `memory.max` (bytes) and `pids.max`; not negative. See [the plugin sandbox](/docs/extending/drivers#the-plugin-sandbox) | A pin is `sha256/` plus the standard base64 of the SHA-256 of a SubjectPublicKeyInfo, the form HPKP and `curl --pinnedpubkey` use. `bcd doctor` warns when the file is readable or writable by anyone but its owner. ### join-encryption.jwk [#join-encryptionjwk] The pins alone cannot seal a token, so the script also writes the JoinService encryption public key as a JWK. `bcd join` uses it only when its SPKI hash equals `join_encryption_key_pin`. When the file is absent and `bcd join --server https://…` names the control plane, bcd fetches `/.well-known/better-compute` (system roots, a 30-second timeout, at most 64 KiB) and uses its `join.encryptionJwk`, again only when it matches the pin. The design's fallback for a stale file is not implemented: a mismatched file, or a missing one without `--server`, fails `bcd join` with exit code 1. See [the join key and its fallback](/docs/guides/nodes/joining#the-join-key-and-its-fallback). An OS that `deploy.image` installed needs neither the script nor this file: the ramdisk writes its own pinned `config.toml` (gateway, server name, site and the three pins) and `join-encryption.jwk` into the installed root, both mode 0600, and the deploy's user data runs `bcd join --server … --token-file /etc/bcd/join-token --node-id …` (with the three pin flags when provision is given `userData.pins`), then `bcd node`. See [profiles and images](/docs/guides/bare-metal/profiles-and-images). ### Joining from the well-known document [#joining-from-the-well-known-document] A machine with no `config.toml` at all, such as an OS that Hetzner Robot's `bcd-join.service` or provision's user data set up with a token, the control plane's URL and three pins, can still join. `bcd join --server https://… --serving-ca-pin … --join-signing-pin … --join-encryption-pin …` reads `/.well-known/better-compute` for what the pins cannot say (where the gateway is), checks the rest against the pins, joins, and writes its own configuration. | Field of the document | Becomes | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `gateway.url` | `gateway_url` | | `gateway.serverName` | `gateway_server_name` | | `gateway.servingCaPin` | compared with `--serving-ca-pin`, which becomes `serving_ca_spki_pin` | | `join.signingKeyPin`, `join.encryptionKeyPin` | compared with `--join-signing-pin` and `--join-encryption-pin`, which become `join_signing_key_pin` and `join_encryption_key_pin` | | `join.encryptionJwk` | the key the token is sealed to, used only when it matches `--join-encryption-pin` | The pins are `sha256/`, as in `config.toml`, and a host build requires all three to bootstrap ("joining without config.toml needs --serving-ca-pin, --join-signing-pin and --join-encryption-pin from the install"). A document whose published pin differs from one given is refused ("refusing to join"). Only a fakehw development build may join with no pins at all, taking them from the document (and it also accepts an `http://` server). On a machine that already has `config.toml`, the configuration wins, and any pin flag given must match it. The control plane publishes the `gateway` section when its options set `gateway: { url, serverName? }`, with `servingCaPin` computed from `pki.servingCa`; the examples read `BETTER_COMPUTE_GATEWAY_URL` and `BETTER_COMPUTE_GATEWAY_SERVER_NAME`. bcd fetches the document over the control plane's own TLS (system roots, a 30-second timeout, at most 64 KiB, redirects refused), refuses a document without `gateway.url` ("install /etc/bcd/config.toml instead"), joins, and only then writes `config.toml` (0600, with the site the join answered) and `join-encryption.jwk` (0644) beside it. Re-running it on a node that already joined exits 0 without a token, so a join unit can run on every boot. The pins come from the sealed PKI bundle: `bootstrapPins(bundle)` in `better-compute/pki` returns all three, and provision's user data (`provisionModule({ userData: { pins } })`) and a Robot bootstrap (`vps({ bootstrap: { pins } })`) render them into the machine; see [profiles and images](/docs/guides/bare-metal/profiles-and-images). > **Why the pins travel with the install.** The well-known document is authenticated only by the control plane's TLS certificate. Taking its pins on trust would be trust on first use: whoever can serve that name with a publicly trusted certificate (a DNS hijack plus a certificate for the name) could hand the machine their own gateway and JoinService keys, open the sealed token and take the node's identity. Pins that come with the install, from the sealed bundle, are stronger, which is why a host build refuses to bootstrap without them. A fakehw build joining with no pins is trust on first use; keep that to development. ## The systemd unit [#the-systemd-unit] By default the script installs `bcd node` running as root, so the unit narrows what root can do. With `--unprivileged` the same unit gets a drop-in that runs it as the `bcd` user beside a root `bcd helper`; see [running bcd node unprivileged](#running-bcd-node-unprivileged). ```ini title="/etc/systemd/system/bcd.service" [Unit] Description=better-compute node agent (bcd) Documentation=https://github.com/better-compute/better-compute After=network-online.target Wants=network-online.target StartLimitIntervalSec=300 StartLimitBurst=10 [Service] Type=simple ExecStart=/usr/local/bin/bcd node --config /etc/bcd/config.toml Restart=always RestartSec=5s UMask=0077 StateDirectory=bcd StateDirectoryMode=0700 RuntimeDirectory=bcd RuntimeDirectoryMode=0700 ConfigurationDirectory=bcd NoNewPrivileges=yes ProtectSystem=strict ProtectHome=yes PrivateTmp=yes ReadWritePaths=-/etc/cdi -/var/run/cdi ProtectKernelModules=yes ProtectControlGroups=yes ProtectClock=yes ProtectHostname=yes RestrictSUIDSGID=yes RestrictRealtime=yes RestrictNamespaces=yes LockPersonality=yes SystemCallArchitectures=native RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK CapabilityBoundingSet=CAP_SYS_BOOT CAP_SYS_ADMIN CAP_SYS_RAWIO CAP_DAC_READ_SEARCH CAP_NET_ADMIN CAP_CHOWN CAP_FOWNER CAP_DAC_OVERRIDE [Install] WantedBy=multi-user.target ``` | Setting | Effect | | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `ProtectSystem=strict` | The whole file system is read-only for bcd, except its state, runtime and configuration directories and the `ReadWritePaths` | | `ReadWritePaths=-/etc/cdi -/var/run/cdi` | The CDI spec directories the devices converger writes; the `-` tolerates a missing directory | | `StateDirectory`, `RuntimeDirectory` | systemd creates `/var/lib/bcd` and `/run/bcd` with mode 0700 | | `NoNewPrivileges`, `RestrictSUIDSGID` | No privilege gain through exec or set-id files | | `ProtectHome`, `PrivateTmp` | No access to home directories; a private `/tmp` | | `ProtectKernelModules`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname` | Kernel modules, cgroups, the clock and the hostname are off limits | | `RestrictAddressFamilies` | Unix sockets, IPv4, IPv6 and netlink (for NIC inventory) only | | `CapabilityBoundingSet` | Rebooting, device and raw I/O access, network administration and file ownership; nothing else | | `Restart=always`, `StartLimitBurst=10` in 300 s | Restarts after 5 s, and gives up after 10 starts in five minutes | `bcd node` opens no TCP listener; everything it does starts with an outbound connection to the gateway. See [privilege separation](/docs/daemon#privilege-separation) for the helper mode. ### The plugin drop-in [#the-plugin-drop-in] The unit above cannot run sandboxed [bcx plugins](/docs/extending/drivers#the-plugin-sandbox): its `CapabilityBoundingSet` has no `CAP_SETUID` or `CAP_SETGID`, which bcd needs to start each plugin under its own uid, and `ProtectControlGroups=yes` makes the cgroup tree read-only. The unit stays that tight; a node that runs plugins gets a drop-in instead. The script installs `/etc/systemd/system/bcd.service.d/plugins.conf` (0644) exactly when `/etc/bcd/config.toml` has a `[plugins]` section, and removes it otherwise, on every run, first install and upgrade alike: ```ini title="/etc/systemd/system/bcd.service.d/plugins.conf (with a privileged plugin)" [Service] # bcx plugins ([plugins] in /etc/bcd/config.toml): their own uids, their own cgroups. CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_KILL Delegate=yes DelegateSubgroup=main ProtectControlGroups=no # plugins.privileged in /etc/bcd/config.toml: runtime plugins that run VMs as root (Kata). CapabilityBoundingSet=CAP_MKNOD CAP_SYS_RESOURCE CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_RAW CAP_SYS_CHROOT CAP_FSETID CAP_SETPCAP CAP_SYS_PTRACE RestrictNamespaces=no RestrictAddressFamilies=AF_VSOCK AF_PACKET ProtectKernelModules=no ``` | Setting | Why | | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `CapabilityBoundingSet=CAP_SETUID CAP_SETGID CAP_KILL` | Added to the unit's set: switch each plugin to its own uid and gid, and stop it. Nothing else is loosened | | `Delegate=yes`, `DelegateSubgroup=main` | systemd hands bcd its cgroup, `system.slice/bcd.service`, and runs bcd itself in the `main` child, so the plugins' cgroup can sit beside it | | `ProtectControlGroups=no` | bcd can create the plugins' cgroups and write their `memory.max` and `pids.max` | The last four lines are added only when `[plugins] privileged` names a plugin (3b7ae79, following d902063): a runtime plugin such as Kata runs VMs as root, so it needs close to root's capabilities (locking VM memory, device nodes, virtiofsd's ptrace), namespaces and mounts, vsock and packet sockets, and the vhost and vsock kernel modules: | Setting | Why | | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `CapabilityBoundingSet=CAP_MKNOD CAP_SYS_RESOURCE CAP_IPC_LOCK CAP_SYS_NICE CAP_NET_RAW CAP_SYS_CHROOT CAP_FSETID CAP_SETPCAP CAP_SYS_PTRACE` | Added to the bounding set for the privileged plugin, which runs as root | | `RestrictNamespaces=no` | The plugin makes namespaces and mounts | | `RestrictAddressFamilies=AF_VSOCK AF_PACKET` | Added to the allowed socket families: vsock to its guests, packet sockets | | `ProtectKernelModules=no` | The vhost and vsock modules | Confined plugins are unchanged: each drops every capability when bcd switches it to its own uid, with `no_new_privs` and the seccomp filter. A node without privileged plugins gets none of these four lines. `--plugins` makes a node a plugin node. The script then creates `/usr/local/lib/bcd/plugins` (root-owned, 0755, so each plugin's uid can reach its binary) and, unless the configuration already has a `[plugins]` section, appends one: ```toml title="/etc/bcd/config.toml (with --plugins --privileged-plugins kata)" [plugins] dir = "/usr/local/lib/bcd/plugins" cgroup = "/sys/fs/cgroup/system.slice/bcd.service/plugins" privileged = "kata" ``` The cgroup is a child of bcd's delegated cgroup. At start bcd creates it and enables the memory and pids controllers in it and in its parent; on systemd older than 254, which ignores `DelegateSubgroup=main`, bcd first moves itself into a `main` leaf, since cgroup v2 gives no controllers to the children of a cgroup that holds processes itself (d902063). If the cgroup cannot be prepared, no plugin runs and the log says why. `--privileged-plugins NAMES` implies `--plugins` and writes `privileged`; each name must be letters, digits, `.`, `_` and `-`. Other plugins you put in the directory yourself, the binaries and their lines in its `pins` file (see [driver plugins in bcd](/docs/extending/drivers#driver-plugins-in-bcd)); the plugins a release ships the script can install (below). A section you wrote by hand works the same way: any `[plugins]` section gets the drop-in, and removing the section removes the drop-in on the next run. #### Installing plugins from the release [#installing-plugins-from-the-release] `--install-plugins NAMES` (adf2ae9) installs bcx plugins that a bcd release ships, today only `bcx-ipmi`, the IPMI BMC driver a site agent hosts (see [IPMI through bcx-ipmi](/docs/guides/bare-metal/bmc#ipmi-through-bcx-ipmi)). It implies `--plugins` and needs `--release-manifest`: the CLI takes each plugin's archive digests, `bcx-ipmi__linux_{amd64,arm64}.tar.gz`, from the verified, signed manifest (`RELEASE_INCOMPLETE` when it lists neither), and refuses any other name. ```bash better-compute daemon install-script --pool pool_site --site fsn1 \ --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \ --bcd-version 0.9.1 --release-manifest dist/manifest.json --release-pins sha256/... \ --install-plugins bcx-ipmi > install-bcd.sh ``` For each plugin the script: 1. downloads `{base-url}{base-path}/downloads///linux-` over https (the same version as bcd) and checks the archive against its pinned SHA-256; 2. refuses an archive that names a path outside it (a leading `/` or a `..`), that has no member named after the plugin, or whose member is not a regular file (a symlink or hard link), and extracts only that member; 3. installs it with mode 0755 as `/usr/local/lib/bcd/plugins/` (written as `.new` and renamed); 4. writes its binary's SHA-256 as its line in the directory's `pins` file (0644), replacing an older line for the same plugin and keeping every other line. A rerun replaces the plugin and its pin without duplicating the line. Like `--plugins`, it is refused with `--unprivileged`. The script itself sets up a node (`bcd node`); a site agent that reads the same `config.toml` finds the plugin in its `[plugins] dir` (see [a site agent](#a-site-agent)). `tests/cli-pki.test.ts` runs the install function in a real `sh` against real archives, including a wrong digest, a symlinked member and a `../` member. ### Running bcd node unprivileged [#running-bcd-node-unprivileged] `bcd helper` (28dc2ae, PLAN.md §6, v0.4) lets `bcd node` run as an unprivileged account while a root process beside it does the privileged work: the CDI specs, allocation env files and sshd files bcd owns, the `bcd_instances` and `bcd_taps` nftables tables, GPU resets, reboots and sshd reloads. [Privilege separation](/docs/daemon#privilege-separation) describes the operations and the checks. The mode is opt-in: generate the script with `--unprivileged` (169c459). ```bash better-compute daemon install-script --pool pool_h100 --site fsn1 \ --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \ --bcd-version 0.1.0 --sha256 amd64= --unprivileged > install-bcd.sh ``` The flag adds a `[helper]` section to `config.toml` unless the file already has one. Both processes read that file: ```toml title="/etc/bcd/config.toml ([helper] section, as --unprivileged writes it)" [helper] socket = "/run/bcd-helper/helper.sock" user = "bcd" ``` | Key | Meaning | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `[helper] socket` | The absolute path of the helper's unix socket. Unset (the default), `bcd node` does its privileged work itself, as root | | `[helper] user` | The account `bcd node` runs as, and the only caller the helper serves besides root; its primary group owns the socket. Required with `socket`, and never `root` | | `[helper] instance_bridges` | On a VPS or sandbox host: the bridges instance NICs may join, which the helper checks every libvirt domain and every `cloud-hypervisor` tap (`ch.taps`, 7fe74a1) against; unset, it defines no domain with a NIC and creates no tap. `--instance-bridges br0,br-priv` adds it | The socket is not in `/run/bcd`: the `bcd` user owns that directory, and a directory the node can write would let it redirect root's work on the socket path with a symlink. So it lives in the helper's own root-owned runtime directory. Since 2d67b25 the helper refuses to start unless the socket's directory is root's and closed to group and other writes, and it binds, chmods and chowns the socket in a private 0700 directory before renaming it into place. Since c139b9c, a helper running as root also refuses to start unless every existing directory from `/` down to the slots directory, the socket's directory and (since 9e71b1f) the entrypoint's directory is root's and closed to group and other writes. It checks this before it touches its update records or arms a watchdog. A host whose `/usr/local/lib` is group-writable (`root:staff` 2775 on older Debian) fails it; `bcd doctor`, run as root on a helper node, reports each such directory as a failed `helper` check. The script decides the mode from the configuration, not the flag: it sets up helper mode exactly when `config.toml` has a `[helper]` section, so an upgrade run keeps the node's mode, and deleting the section returns the node to root on the next run (the drop-in is removed; the account and `bcd-helper.service` are left, and the helper cannot start without the section, so disable that unit yourself). In helper mode it: 1. creates the `bcd` system account if there is none (`useradd --system --no-create-home --home-dir /var/lib/bcd --shell /usr/sbin/nologin bcd`, or busybox `adduser -S -D -H` without `useradd`); 2. installs `bcd-helper.service` and the `bcd.service` drop-in `unprivileged.conf`, both 0644; 3. creates `/usr/local/lib/bcd`, where the helper keeps the self-update slots, with mode 0755, and sets an existing one to 0755 (27c9551). The helper's `UMask=0077` would otherwise leave it 0700, and `bcd node`, running as `bcd`, could not execute the slot its entrypoint links to after the first update. The helper itself creates every missing directory on the way to the slots with mode 0755 whatever its umask, and never changes the mode of one that exists (c139b9c, 9e71b1f); 4. makes `/etc/bcd` (0750) and `config.toml` (0640) group-owned by `bcd`, so the node can read its configuration and nothing else of `/etc/bcd` becomes readable to it. Both stay root's: since 3fa9021 `bcd helper` refuses to start as root unless its `config.toml` is a regular file root owns that no one else can write, in directories only root can write, because that file (`[host] units`, `command_root_sha256`, the join pin) decides what root does for the node. Do not hand the file to `bcd` or make it group-writable; 5. on a host with `/etc/libvirt/qemu.conf` (d418820): sets `dynamic_ownership = 0` there (replacing the line, commented or not, or appending it), adds `libvirt-qemu` or `qemu`, whichever exists, to the `bcd` group, and restarts `virtqemud.service` or `libvirtd.service` if it runs. libvirt then chowns no path the node names, and QEMU reaches the node's disk files through the `bcd` group (see [libvirt through the helper](/docs/daemon#libvirt-through-the-helper)); 6. keeps the `bcd` user away from the TPM (8870b8d, security review 3). The kernel's resource manager filters no command, so any member of the `tss` group could change the hierarchies' authorization, clear the TPM or trip its dictionary-attack lockout, breaking root's TPM-sealed secrets (LUKS keys) and attestation. `bcd node` therefore never opens `/dev/tpmrm0`: `bcd helper` derives, signs with and certifies its node keys (see [node keys through the helper](/docs/daemon#node-keys-through-the-helper)). Scripts from 0ba2a79 until then added `bcd` to `tss` and wrote a udev rule; an upgrade run takes `bcd` out of `tss` when it is a member (`gpasswd -d bcd tss`, or busybox `delgroup bcd tss`) and deletes `/etc/udev/rules.d/60-bcd-tpm.rules`. It does not change the device's current group or mode back. Until that run, a host installed between 0ba2a79 and 8870b8d keeps `bcd` in `tss`, and since e6c3f53 `bcd doctor` fails its `tpm` check there: "bcd is in the tss group: it can send the TPM any command (lock or clear its hierarchies); bcd helper holds the node keys, so remove it (gpasswd -d bcd tss) and restart bcd.service". Rerun the script, or do that by hand; 7. after `bcd join` (which runs as root), gives `/var/lib/bcd` to `bcd:bcd` and starts `bcd-helper.service` before `bcd.service`; an upgrade run does the same with restarts. ```ini title="/etc/systemd/system/bcd-helper.service" [Unit] Description=better-compute node agent privileged helper (bcd helper) Documentation=https://github.com/better-compute/better-compute Before=bcd.service [Service] Type=simple ExecStart=/usr/local/bin/bcd helper --config /etc/bcd/config.toml Restart=always RestartSec=2s UMask=0077 RuntimeDirectory=bcd-helper RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target ``` ```ini title="/etc/systemd/system/bcd.service.d/unprivileged.conf" [Unit] Requires=bcd-helper.service After=bcd-helper.service [Service] User=bcd Group=bcd SupplementaryGroups=systemd-journal adm ``` | Setting | Why | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `Requires=`, `After=bcd-helper.service` | The node starts after the helper and stops with it | | `User=bcd`, `Group=bcd` | `bcd node` runs unprivileged; systemd hands it the unit's state and runtime directories (`/var/lib/bcd`, `/run/bcd`) | | `SupplementaryGroups=systemd-journal adm` | Read access to the system journal and logs (`logs.collect` runs `journalctl`) | | `RuntimeDirectory=bcd-helper`, mode 0755 | `/run/bcd-helper`, root-owned, holds the socket (0660, `root:bcd`) | The helper serves a caller only when `SO_PEERCRED` shows `bcd`'s uid and gid (or root) and `/proc//exe` is an installed bcd: `[update] entrypoint`, `bcd-a` or `bcd-b` in `[update] slots_dir`, or the helper's own binary. A node run from anywhere else is refused. Self-update works in this mode (2d67b25): the node downloads a release into `/staging`, and the helper verifies it under its own `[update]` pins and floor, copies it into root's `[update] slots_dir` (default `/usr/local/lib/bcd`) and swaps; see [self-update through the helper](/docs/daemon#self-update-through-the-helper). Nodes installed by a script generated with `--release-pins` get the same `[update]` section as a root node. > **Plugins need root.** `--unprivileged` together with `--plugins` or `--privileged-plugins` is a usage error ("--unprivileged cannot host bcx plugins yet: bcd node would need root to start them"). What the helper does not do runs with the `bcd` account's own rights: the journal and the node key file (on a host with a TPM only the key's record; the key itself is used through the helper's `tpm.*` operations), the status socket, the `libvirt-kvm` runtime's reads (`virsh -r`, libvirt's read-only socket) and its files under `data_dir/libvirt`, the `cloud-hypervisor` runtime's files, which it shares with each VMM's user through ACL entries, and reading `/dev/kmsg`. Defining and running libvirt domains goes through the helper's checked `libvirt.*` operations (ed3c7d2; see [libvirt through the helper](/docs/daemon#libvirt-through-the-helper)), and `cloud-hypervisor`'s taps, VFIO groups and VMM units through its `ch.*` operations (7fe74a1; see [cloud-hypervisor through the helper](/docs/daemon#cloud-hypervisor-through-the-helper)). Only `bcd node` uses the helper; `bcd site` ignores `[helper]`. `tests/e2e/bcd.test.ts` runs both processes by hand on a fakehw build, a GPU hold and a signed self-update included; `tests/cli-pki.test.ts` checks the script's section, account, units and drop-in. ## Joining [#joining] The join itself is `bcd join`, which the script runs for you: 1. bcd generates a P-256 key and a CSR. On a host with a TPM the key is made in the TPM (50e18f9) and `node.key` (0600) under `data_dir` keeps only its record; without one, or when the TPM cannot hold the key, `node.key` is a PKCS#8 file. See [the node key in the TPM](/docs/guides/nodes/attestation#the-node-key-in-the-tpm). 2. It seals the token to the JoinService encryption key and calls `Join` on the gateway over TLS pinned to the serving CA, with its hardware claims and the attestor `join-token`. The gateway relays the call; it can neither read the token nor forge the answer. A deferral is retried with the same CSR; retryable errors back off. 3. It verifies the signed `JoinResponse` under the pinned signing key, checks that the response is for its own CSR, that the leaf certifies its key, the SPIFFE id and the role, and stores the certificate chain and trust bundle in `identity.json`. A journal left by an earlier identity is moved aside. bcd never takes the token from a command-line argument, so a token pasted in the wrong place does not reach shell history or logs. `bcd join` rejects positional arguments without echoing them: ```bash $ bcd join bcjt_jt_01... bcd join: takes no positional arguments bcd join reads the join token from BCD_JOIN_TOKEN or stdin, never from arguments ``` `--token-file PATH` reads the token from a file instead (c29bb16). The file must be its owner's alone: mode 0600 or 0400 (a Kubernetes Secret volume with `defaultMode: 0400`, whose `..data` symlink bcd follows), a regular file, and at most 4 KiB; anything else is refused before the token is read. Once the join succeeds, or when the machine had already joined, bcd deletes the file, since a token is single use; on a read-only mount such as a Secret volume it cannot, and logs a warning instead of failing. [bcd on Kubernetes](/docs/daemon/kubernetes) joins this way. The node then opens its session (see [sessions](/docs/daemon/sessions)) and appears as `adopting`. Tokens, attestors, refusals and certificates are covered in [joining nodes](/docs/guides/nodes/joining). ### Joining again with --reset [#joining-again-with---reset] `bcd join` on a machine whose identity loads answers `already joined as ` and exits 0, whatever state the certificate is in: expired, revoked, refused at renewal, or naming an AK this TPM no longer has. That keeps a join unit safe to run on every boot, but it cannot bring such a node back. Since a1c1a46 `--reset` can: ```bash # as root: see what it would move (moves nothing, exits 2) bcd join --config /etc/bcd/config.toml --reset # move the identity aside and join again, here with the TPM bcd join --config /etc/bcd/config.toml --reset --yes --attestor tpm # or with a new join token on standard input bcd join --config /etc/bcd/config.toml --reset --yes < join-token.txt ``` | Flag | Does | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--reset` | Before the join, moves the node's identity and journal aside: `identity.json`, `node.key`, a pending `node.key.next` and `journal.db`, whichever exist in `data_dir`, each renamed `.reset-` (for example `node.key.reset-20260926T114200Z`) | | `--yes` | Confirms `--reset`. Without it bcd prints which files it would move, that the join that follows makes a new key (a new node, unless the attestor proves this machine and reclaims it), and exits 2 | What it does and does not do: * **Root only.** Outside development (fakehw) builds a non-root `--reset` fails with "run it as root", exit 1. * **Nothing is deleted.** The old files stay in `data_dir` under their new names, so an operator can still read them (a TPM key's record still names its key). Remove them yourself once the node is back. * **Then the usual join.** The move happens after the configuration loads and before the join reads any credential, so the same run needs the join's credential as any join: a token (`BCD_JOIN_TOKEN`, standard input or `--token-file`), `--attestor tpm`, or `--node-id` for an installed OS. If the join then fails, the identity stays moved aside and a later plain `bcd join` joins. bcd's own hints spell it as two runs, `bcd join --reset --yes`, then `bcd join`. With nothing to move, `--reset` does nothing and the join goes on. * **Same node or a new one.** An attestor that proves the machine (`tpm` with the same EK, `provisioned`) reclaims the node it is bound to. A token join of a machine whose trusted hardware key the old node still holds needs a token with `allowReclaim` (see [placeholder hardware and reclaims](/docs/guides/nodes/joining#placeholder-hardware-and-reclaims)); to join as a new node instead, cordon and delete the old one first (`better-compute nodes delete --yes`; see [removing an adopted host](/docs/guides/nodes/operating-nodes#removing-an-adopted-host)). * **The journal moves too** (b4fcc2e). `journal.db` is the old node's, and `bcd node` refuses a journal of another node ("the journal belongs to node ``, not ``"). A plain join moves an old journal aside only when the old identity still loads, which after `--reset` it does not, so `--reset` moves it with the identity and the node starts a new one. * **Under `[helper]`** the join runs as root and writes the new `node.key` and `identity.json` as root. When it finishes, it hands both to the `[helper]` user with `Lchown` (b4fcc2e; root only, and a name swapped for a symlink changes only the link), for a new join and for `already joined` alike, so `bcd node` can read its key. Then restart `bcd-helper.service` and `bcd.service`; re-running the install script also does the restarts. `bcd doctor` points at it: the `identity` check fails for an identity that does not load and for an expired certificate, and both details end with "bcd join --reset --yes, then bcd join". `TestJoinReset` covers the dry run, the move (the journal included) and that no identity loads afterwards; `TestJoinHandsTheIdentityToTheNode` covers the hand-over (as root; skipped otherwise). ## Site agents and the ramdisk [#site-agents-and-the-ramdisk] The install script sets up nodes only: it joins with role `node` and its unit runs `bcd node`. The other two agent modes use the same binary but are set up differently. ### A site agent [#a-site-agent] A site agent reads the same `config.toml` and `join-encryption.jwk`, joins with a [site token](/docs/guides/bare-metal/sites#enroll-site-agents-with-site-tokens) and runs `bcd site`: ```bash BCD_JOIN_TOKEN=bcjt_... bcd join --role site --config /etc/bcd/config.toml bcd site --config /etc/bcd/config.toml ``` `bcd site` refuses a data directory whose identity has another role. Without a `[boot]` section it runs BMC operations, `site.httpProxy` and boot entries but serves no PXE machine (it logs a warning). With one, it also runs the boot server: ```toml title="/etc/bcd/config.toml (site agent)" gateway_url = "https://gw.eu.example.com" site = "site_01k5w..." serving_ca_spki_pin = "sha256/..." join_signing_key_pin = "sha256/..." join_encryption_key_pin = "sha256/..." [boot] listen = "0.0.0.0:8080" url = "http://10.20.0.2:8080" kernel = "http://10.20.0.2:8080/ramdisk/vmlinuz" initrd = "http://10.20.0.2:8080/ramdisk/initrd" cmdline = "console=ttyS0,115200" server_ip = "10.20.0.2" proxy_dhcp = true tftp = ":69" tftp_root = "/srv/bcd/ipxe" ``` | `[boot]` key | Meaning | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `listen` | The HTTP address of the boot server; empty turns the boot server off. Without `join-encryption.jwk` beside the config `bcd site` exits 1, and without `kernel` and `initrd` the boot server stops with an error | | `url` | How machines reach the boot server. iPXE chains to it, and the ramdisks it boots get `/cache` as their image cache | | `kernel`, `initrd` | The URLs of the site's ramdisk kernel and initramfs, which iPXE fetches | | `cmdline` | Appended to every ramdisk's kernel command line | | `server_ip` | The agent's IPv4 address on the provisioning network: the proxy DHCP server id and TFTP next-server | | `proxy_dhcp` | `true` answers PXE and UEFI HTTP boot clients on UDP 67 and 4011, beside the site's own DHCP server, which keeps handing out addresses | | `tftp`, `tftp_root` | The TFTP server's UDP address and the directory of iPXE builds it serves: `undionly.kpxe` (BIOS), `ipxe.efi` (x86-64 UEFI), `ipxe-arm64.efi` (arm64 UEFI) | | `cache_max_bytes` | The image cache's byte budget (0, the default, is unbounded): least recently used images are evicted first, never one the `site` section's `cache_images` still names | `proxy_dhcp` and `tftp` need `server_ip` and `url`, or `bcd site` exits 2. The iPXE builds and the ramdisk kernel and initramfs are yours to supply: the repository builds neither. Ports 67, 69 and 4011 are privileged, and the unit the install script writes keeps `bcd` to a capability set without `CAP_NET_BIND_SERVICE`, so a site agent needs its own unit. What the boot server answers, and when, is on [sites](/docs/guides/bare-metal/sites#the-boot-server-in-bcd-site). ### The ramdisk [#the-ramdisk] `bcd ramdisk` is not installed. It runs inside the PXE ramdisk the site agent boots, and takes everything from the kernel command line the site agent rendered into the machine's iPXE script, so it has no `config.toml`: | Flag | Default | Meaning | | ------------ | --------------- | ----------------------------------------------------------------------------- | | `--cmdline` | `/proc/cmdline` | Where to read the `bc.*` boot parameters | | `--data-dir` | `/run/bcd` | The key, identity and journal, on tmpfs: every ramdisk boot is a new identity | | Boot parameter | Meaning | | ---------------- | ----------------------------------------------------------------------------------------- | | `bc.gateway` | The gateway URL (an `http://` one only in fakehw builds) | | `bc.server_name` | Optional: the name the gateway certificate carries | | `bc.cache` | Optional: the site image cache, tried before an image's source | | `bc.site` | The site id | | `bc.pins` | The serving CA, join signing and join encryption pins, comma-separated | | `bc.jwk` | The JoinService encryption key's public JWK, base64url; refused unless it matches its pin | | `bc.token` | The machine's single-use, MAC-bound `discovery` join token | | `bc.assertion` | The site agent's `SignedDiscoveryAssertion`, base64url | A missing parameter exits 2. The ramdisk joins with the `discovery` attestor as role `ramdisk` (see [attestation](/docs/guides/nodes/attestation#discovery)), logs JSON, then runs the agent until the machine kexecs into its installed OS or powers off. Its image needs `bcd` plus the tools its kinds call: `nvme-cli`, `hdparm`, util-linux (`blkdiscard`, `blockdev`, `lsblk`, `mount`), e2fsprogs' `debugfs` and `mtools` (to write into and read from a deployed image's ext and FAT filesystems without mounting them; 1ae1789, d9a900f), `udevadm`, `kexec-tools` and, for `diag.dcgm`, `dcgmi`. ## Upgrades [#upgrades] ### Before v0.4: rerun the script [#before-v04-rerun-the-script] Run the same script, or one generated for a newer `--bcd-version`, without a token on a host that already has `/etc/bcd/config.toml`. It downloads and verifies the new binary, replaces `/usr/local/bin/bcd` with an atomic rename, rewrites the key, the unit and the top-level keys of the configuration, and restarts the service: ```bash sudo sh install-bcd.sh # bcd upgraded to 0.1.1 and restarted. ``` The configuration's top-level keys (gateway, site and the three pins) come from the new script, and every section of the existing file (`[plugins]`, `[boot]`, `[imds]`, `[journal]`, `[log]`, ...) is kept as it was (4225772; earlier scripts rewrote the file whole and dropped them). Top-level keys you added by hand, such as `data_dir` or `gateway_server_name`, are not kept, so put them back after an upgrade or regenerate the script. The plugin drop-in follows the kept configuration: a node with `[plugins]` keeps it, even when the new script was generated without `--plugins`. The node keeps its identity and journal across the restart. On reconnect its `Hello` carries the new `daemon_version`, which the gateway records in `node_sessions.daemon_version`. On a node that has already [self-updated](#self-update), the script's `mv -f` replaces the entrypoint symlink with a plain file. The next self-update adopts that file into the current slot before it relinks the entrypoint (9006eee), so it never falls back to the older binary the slot held; before that fix it did, rolling the script's upgrade back. ### Upgrading an unprivileged libvirt host [#upgrading-an-unprivileged-libvirt-host] A VPS or sandbox host installed with `--unprivileged` by a script from before d418820 runs `bcd node` in the `libvirt` group, which is root-equivalent, and its libvirt still relabels files. A new bcd there sends domain operations through the helper anyway (ed3c7d2), which refuses to define a domain with a NIC until `[helper] instance_bridges` is set, but the group membership and the relabelling stay until you run a newer script, generated with `--unprivileged` (and `--instance-bridges` for the instances' bridges), on the host without a token. It rewrites the `unprivileged.conf` drop-in without `SupplementaryGroups=libvirt`, sets `dynamic_ownership = 0`, adds QEMU's user to the `bcd` group, restarts libvirt and then both bcd units: ```bash better-compute daemon install-script --pool pool_vps --site fsn1 \ --gateway-url https://gw.eu.example.com --base-url https://compute.example.com \ --bcd-version 0.9.2 --sha256 amd64= --unprivileged --instance-bridges br0 > install-bcd.sh sudo sh install-bcd.sh ``` Without `[helper] instance_bridges` the helper defines no domain with a NIC, so set the bridges before instances with networks start there; the script adds them only when the configuration has no `instance_bridges` line yet. `bcd doctor`'s `libvirt` check fails until the host is set up this way. ### From v0.4: rollouts and self-update [#from-v04-rollouts-and-self-update] The v0.4 design moves upgrades into desired state. A [rollout](/docs/guides/rollouts) sets `NodeSpec.daemon = { version, url, sha256, signature }` on nodes wave by wave (1%, 10%, 50%, 100% by default) within each pool's disruption budget, and a node succeeds when it rejoins reporting that version. A daemon too old to speak the server's minimum protocol version gets `Welcome.upgrade_required`, and its session accepts only the `daemon.upgrade` command kind, which carries the same `DaemonSection`; the rollout controller sends it to such nodes. ### Self-update [#self-update] bcd updates itself in place (de9f380, `daemon/internal/agent/update.go`, `daemon/internal/selfupdate`), for both the `daemon` section and `daemon.upgrade` (node and site roles). Since 2d67b25 the privileged half (`selfupdate.Manager`: verify, swap, confirm, roll back) runs in process on a root node and inside `bcd helper` on an [unprivileged one](#running-bcd-node-unprivileged); see [self-update through the helper](/docs/daemon#self-update-through-the-helper). The steps are the same: 1. **Only a signed release.** `DaemonSection.signature` is the release's signed manifest (`manifest.json` from `better-compute release sign`). At least `[update] release_threshold` distinct keys pinned in `[update] release_pins` must have signed it (purpose `release-manifest`), and it must be for product `bcd` and the section's version, have a sequence not below the release this node ran or was installed at (`[update] release_sequence`; no rollback), be unexpired on both the host clock and server time, and list the bare binary `bcd___` with the section's SHA-256. Without release pins bcd takes no binary at all: the control plane alone can never hand a node one. The URL must be `https://`. 2. **Two slots.** The binaries live in `bcd-a` and `bcd-b` of the slots directory (`[update] slots_dir`: default `/bin`, or `/usr/local/lib/bcd` under the helper), with `current` and `previous` symlinks and `slots.json`. The first update adopts the binary the install script put at the entrypoint (`[update] entrypoint`, default `/usr/local/bin/bcd`) into slot `a` and makes the entrypoint a symlink to `current`; an entrypoint that is a symlink somewhere else is refused, and a plain file found there later (a rerun of the install script) is copied into the current slot first (9006eee). Running as root, adoption takes only a root-owned regular file that group and others cannot write (9e71b1f). The node downloads the release into `/staging/bcd-`, at most 512 MiB, checked against the manifest's size and digest as it streams, with an idle watchdog; the swap then copies it into the other slot, checking the size and digest again as it writes. It opens the staged file beneath the staging directory's parent (`os.Root`), refuses a symlinked staging directory, opens with `O_NOFOLLOW|O_NONBLOCK` (a FIFO cannot hang it), and takes only a regular file with one link; under the helper the file must also be owned by the node's uid (c139b9c). 3. **Swap and confirm.** bcd writes `upgrade.json` with a deadline of now plus 120 seconds (fsynced), points `current` at the new slot and execs the entrypoint once everything is closed. The new binary confirms itself on its first `Welcome`: the release floor moves up (`update.json`) and `upgrade.json` goes. Without a `Welcome` by the deadline it swaps back and restarts; the old binary then records the version as failed ("bcd X did not start"), reports the `daemon` section `FAILED` with reason `UpdateFailed`, and never tries that version again until `bcd update --clear-failed`. Under the helper, the node asks the helper to confirm, and the helper's own watchdog also swaps back an update still unconfirmed 30 seconds after its deadline and restarts `bcd.service`. 4. **Crash loops.** Under systemd the first update writes `bcd-rollback.service` and a `bcd.service` drop-in (`50-bcd-rollback.conf`: `StartLimitIntervalSec=600`, `StartLimitBurst=5`, `OnFailure=bcd-rollback.service`), which restores `previous`, only while `upgrade.json` exists. fakehw builds never write them. 5. **No going back below the floor.** A confirmed update is never undone (c139b9c). `slots.json` records each slot's version, manifest sequence and the SHA-256 of the bytes written into it. The binary already in the other slot is taken without a manifest only when its recorded sequence is known and not below the release floor, so a section that names the older version the node ran before is refused: without a manifest there is nothing to verify, and that release's manifest is below the floor (the `daemon` section reports `FAILED`). An update that never confirmed is still swapped back by the deadline, the crash-loop unit or the helper's watchdog. To undo a release that confirmed, roll forward: sign a newer release with a higher sequence and roll it out ([rollouts](/docs/guides/rollouts#pause-resume-abort)). `daemon.upgrade` is self-terminating, like `node.reboot`: it reports `RUNNING` before the swap and `SUCCEEDED` from the new binary once it is welcomed, or `FAILED` with `update-failed` after a swap back; a restart without a reboot is decided by `upgrade.json`, not the boot id. Running workloads are not touched. `bcd sim` and `bcd ramdisk` run without self-update: the `daemon` section stays `SectionUnsupported` and the kind `UNSUPPORTED`. `bcd update` shows the slots, the release floor, a pending update and failed versions from `[update] slots_dir` (run it as root on a helper node, whose records are root's), and `bcd doctor` warns without release pins, fails when the slots are in use but the entrypoint does not link to `current`, and warns about a pending update or a failed version (2845f41). Both find the slots and records the same way (8eca2b1): `[update] slots_dir`, else `/usr/local/lib/bcd` under the helper, else `/bin` with the records in `data_dir`. `tests/e2e/bcd.test.ts` updates a real bcd to a second signed build through `daemon.upgrade`, a real exec and its `Welcome` (fc0ca6a), and delivers a signed build to an unprivileged node through `bcd helper` (2d67b25); no release has been cut, so no real host has self-updated yet. See [delivery](/docs/daemon/delivery). The fleet-wide order for any upgrade is: expand migrations, then workers and api, then gateways one at a time with `Drain`, then daemons in waves, and contract migrations in the next release. See [version skew](/docs/daemon/protocol#version-skew). ## Removing bcd [#removing-bcd] Neither the CLI nor bcd has an uninstall step, and PLAN.md designs none. Retire the node in the control plane first so nothing is scheduled on it (see [operating nodes](/docs/guides/nodes/operating-nodes)). Then stop and disable `bcd.service` and remove the files listed above. ## Release artifacts [#release-artifacts] bcd is configured to build with goreleaser from `daemon/` (`.goreleaser.yaml`) for Linux on amd64 and arm64, with cgo and `zig cc`, set up for a go-nvml backend that would load `libnvidia-ml`; the GPU code in the tree today reads `nvidia-smi` and needs no cgo (cb987e5). A release produces `bcd__linux_{amd64,arm64}.tar.gz`, each with a `.sha256`, and the bare binaries `bcd__linux_{amd64,arm64}`. The bare binary is what the install script downloads, what a signed release manifest lists and what self-update verifies. The same release builds `bcx-ipmi`, the site agent's IPMI BMC plugin, without cgo and ships it only as `bcx-ipmi__linux_{amd64,arm64}.tar.gz` (348ff56), so a release still has exactly one bare binary per platform, bcd's, which rollouts and self-update take. goreleaser also writes an SPDX SBOM per archive (syft) and a keyless cosign signature and certificate for every artifact. `.github/workflows/release.yml` runs it on a `v*` tag (or a dispatch naming one), appends everything to that tag's GitHub release, and adds SLSA level 3 provenance (`bcd.intoto.jsonl`) from the generic SLSA generator. No tag has been released yet; CI (`.github/workflows/daemon.yml`) vets, tests and runs `goreleaser check`, and a local snapshot build is `goreleaser release --snapshot --clean --skip=sign,sbom`. Until then a binary comes from a plain `go build` in `daemon/`. The development and test builds are cross-compiled without cgo (`GOOS=linux GOARCH=amd64 CGO_ENABLED=0`); `node scripts/daemon-test.mjs --build-bcd` writes the fake hardware build to `daemon/bin/bcd-linux-amd64`. go-nvml is not a dependency, so a cgo build gains nothing today: the default build reads GPUs through `nvidia-smi` without cgo (cb987e5).